What is a Distributed System?

Put quite simply, a distributed system is a number of pieces of hardware and software that communicate with each other, but use different processors. They can communicate over a network such as Local Area Network (LAN), or the Internet (WAN). Distributed systems are subject to a number of problems, however, such as variable latency – which is when 2 identical messages sent between 2 identical computers may now take the same amount of time to arrive. This can cause unpredictable failures, such as messages not being sent.

Definition of a Distributed System

The definition of a distributed system is one which has independent and self-sufficient – often heterogeneous or autonomous – spatially-separated components which must us a common interconnect to exchange information in order to coordinate information, and to make the whole system appear to its user as a single coherent system.

            To further this more a distributed system is the result of collaboration between separate, independent processes. In order for separate, independent processes to collaborate, then they must interact.

            If this interaction is by means of message passing then each component must be aware of the mechanisms for interprocess communication (IPC), as well as the protocols for using these mechanisms.

This definition is described over a number of posts under the category “Distributed Systems”: http://badgerati.wordpress.com/category/revision/distributed-systems/

Mediated Message Exchange (MME)

 This is an architecture that arises from Distributed Systems architectures and event synchronization. Within an MME, Interprocess Communication (IPC) is not direct, unlike in Direct Message Exchange (DME) Architecture. This means that processes do not send and receive data directly; this is because they first have to communicate directly with a Middleware component.

This form of Middleware is referred to as Message-Oriented Middleware – or MOM for short.

The MOM component is interposed between clients and it is used to mediate IPC. It also allows event synchronization to be decoupled in time without threading or forking.

Message-Oriented Middleware

Messages are sent to the MOM in a queue, which then sends the messages to the required receiver client. If a message comes from one source client and goes to one destination client, then the MME Architecture is said to be Point-to-Point.

In some MME Architectures, a message can go to more than one destination client, such as publish-subscribe systems.

In this case, a client subscribes with the MOM expressing an interest in certain events from other clients. Then, when a client publishes an event with the MOM, it notifies the subscribing clients.

Direct Messaging Exchange (DME)

            This is an architecture that arises from Distributed Systems architectures and event synchronization. Because of the way DME’s are built, Interprocess Communication (IPC) follows a very simple pattern.

In DME Architectures, a process must be in a position to receive a message/data from some other process.  Therefore, a process can send data to some other process, and when this other process receives said data, it may process the sent data, and then send a response message to the process the data initially came from.

1. P₂ is in the receiving state
2. P₁ sends some data D₁ to process P₂
3. P₁ is now in the receiving state
4. P₂ receives data D₁ and processes it
5. P₂ now sends a response message D₂ back to process P₁
6. Repeat steps 1-6

Because of the way that a DME Architecture is set up, the event synchronization is unmediated. For example, the interacting processes must themselves take case of synchronizing the events in order to send the data. Because of this, there a 2 types of DME’s:

• Asymmetric
• Symmetric

Asymmetric DME

In an Asymmetric DME, event synchronization is a lot simpler, because one process – say the servers – can just wait for requests from some other processes – say clients. The clients also have a function, once they send a message; they must wait for a response from the server.

Asymmetric DME’s also allow a process to be assigned a special role; for example, a process could act as a server to other processes acting as clients.

So, we say that a client sends a request message to the server, and then waits in receiving mode. However, the server has been waiting in receiving mode for a request message from the client. So, on receiving this message, the server processes it and sends the client a response message.

Because of the structure of an Asymmetric DME, it is common for the server to take on heavier loads. However, this requires balancing, and can prevent scaling.

Typical examples of daily used Asymmetric DME’s are the Web and Email.

asymmetric intefconnect

Symmetric DME

In a symmetric DME the problem is much less acute. No special role is assigned to any process in a symmetric DME, unlike in an asymmetric DME. This time processes are peers, hence the term peer-to-peer (P2P) architecture. P2P architectures are seen as leading naturally to balanced loads, and graceful scaling.

No process simply waits for a request message in order to respond, and no process simply makes a request message and waits for a response. This is why event synchronization is less simple in a symmetric DME.

An example is Skype:

• Skype is a P2P VoIP (Voice over IP) application
• Skype uses super-peers which are basically just peers which take on a specialized role, in order to form a hierarchy amongst peers.
• Skype is therefore not a pure symmetric DME.

Symmetric Interconnect

Ive heard of a teacher getting the date wrong on the whiteboard wrong once in a while. But an examiner getting it wrong? This just takes the biscuit…

A better Example

Let’s take a look at another example, this time lets use:

_n          

∑ i²   =     (n(n+1)(2n+1)) / 6

ⁱ⁼⁰           

Where we have:

                _n

L(n) =    ∑ i²                          and                        R(n) =    (n(n+1)(2n+1)) / 6

                ⁱ⁼⁰

For now I will skip the base case. However, do note that when n=0 for the base case, L(0)=R(0), so that the base case is true. As for the step cases, we have:

• ∑ i²  =  ∑ (n+1)²

Once this is done, we then need to move everything next to the ‘∑’ onto the right hand side. So we get this:

• before we move we have:
             R(n)     =          (n(n+1)(2n+1)) / 6
• Once the ‘i’ has be moved over, then R(n) becomes R(n+1), and we have:
             R(n+1) =          [(n(n+1)(2n+1)) / 6]   +  (n+1)²

Once we have this, it is just simple arithmetic:

(1) = [(n(n+1)(2n+1)) / 6]  +  (n+1)²

(2) = [(n(n+1)(2n+1)) / 6]  +  [6(n+1)² / 6]               make both have same denominator.

(3) = (n(n+1)(2n+1) + 6(n+1)²) / 6                             bring both together

(4) = ((n+1)(n+2)(2n+3)) / 6                                  

(5) = R(n+1)

As from before, step (4) requires you to expand the brackets from step (3), the fractionise the whole lot to obtain (n+1)(n+2)(2n+3).

[ A = for all.    E = there exists ]

The Principles of Induction

Firstly, welcome to Induction. The one thing in Discrete Maths that I hear a heck of a lot of people complaining about! – So don’t worry, you’re not the only one

Here I shall attempt – attempt being the key word – to make induction make a lot more sense to you.

OK, so let’s say we have a set, ‘X’ of natural numbers:

• X = {1, 2, 3, 4…}

The first thing we need to understand about induction is that we have 2 cases.

• First there is the ‘base’ case which is the lowest value of say ‘n’ possible, that makes a given equation true – usually 0.
• Secondly we have the ‘step’ case. This states that if the equation is true for ‘n’, then theoretically it must be true for ‘n+1’.

We can show this relation with set theory, so:

• Base case:      0 Є X
• Step case:       An Є N, n Є X  =>  (n+1) Є N

Let’s say that we have the some function P(n). The base and step cases for this function would be:

• Base:  P(0) = true
• Step:   An Є N, if P(n)=true then so must P(n+1)

Induction

Let’s now have a proper example on how we do induction. Say for example you have the following summation – and you are given the answer:

_n          

∑ i   =     ( n(n+1) ) / 2

ⁱ⁼⁰                

Now that we have this, we need to split up the formula:

                _n

L(n) =    ∑ i                            and                        R(n) =    ( (n(n+1) ) / 2

                ⁱ⁼⁰

For this, we need to say that An Є N, then L(n)=R(n).

For this summation, the Base case is clearly 0. so is it true that L(0)=R(0)? Let’s find out:

• When i=0 in ∑ i, then it is quite obvious that ∑ i=0. Therefore L(0)=0
• In the case if R(0), we have:                = ( (n(n+1)) / 2
                  = ( (0(0+1)) / 2
                  = (0(1)) / 2
                  = 0 / 2
                  = 0 

  Therefore, R(0)=0 as well. So:

• L(0) = R(0) is true. And therefore the Base case is true.

Now we need to move onto the Step case part. This part is a little trickier to explain, so you’ll have to bear with me on this…

We know that L(n)=R(n), however, we need to prove that L(n+1)=R(n+1).

A quick way to do this, is that when we state L(n+1), change all the ‘i’ next to the ‘∑’ to ‘n+1’. So in this case we have: (better example later)

• ∑ i  =  ∑ (n+1)

Once this is done, we then need to move everything next to the ‘∑’ onto the right hand side. So we get this:

• before we move we have:
             R(n)     =          (n(n+1)) / 2
• Once the ‘i’ has be moved over, then R(n) becomes R(n+1), and we have:
             R(n+1) =          [(n(n+1)) / 2]   +  (n+1)

Once we have this, it is just simple arithmetic:

(1) = [(n(n+1)) / 2]  +  (n+1)

(2) = [(n(n+1)) / 2]  +  [2(n+1) / 2]             make both have same denominator.

(3) = (n(n+1) + 2(n+1)) / 2                         bring both together

(4) = ((n+1)(n+2)) / 2                                 This step will be explained further in the sec

(5) = R(n+1)            this step requires some common sense really. Go back to R(n),                                      then sub in (n+1) into all n, from (n(n+1)) / 2. What do you get?

                              Exactly , you should hopefully end up with ((n+1)(n+2)) / 2. if not,                              here:

                              (n(n+1)) / 2

                              ( (n+1) ((n+1)+1) ) / 2

                              ((n+1)(n+2)) / 2

OK then, so, step (4). For this step, you need to expand the brackets. Once that is do you should have: n²+3n+2. Then fractionise this to get: (n+1)(n+2).

Let’s assume that we have 7 nodes:

At this current moment in time, node ‘7’ is the coordinator, because it is the highest numbered node. But then node ‘7’ leaves, so who is now the coordinator? Lets say node ‘4’ decides it wants to be the coordinator, then we have:

Here, node ‘4’ has sent a ELECTION to nodes ‘5’, ‘6’ and ‘7’, this is because none of the nodes know that node ‘7’ has left yet. In this case, node ‘7’ cannot reply, yet, nodes ‘5’, and ‘6’ are still here, so they reply to ‘4’ that they are still here:

Now we have nodes ‘5’ and ‘6’ sending out elections. In this case, node ‘5’ sends an election to node ‘6’ and ‘7’. Whereas node ‘6’ just sends an election to node ‘7’:

Now, since node ‘6’ is still here, then node ‘6’ tells node ‘5’ that it is still here:

Finally, since node ‘7’ has not replied to node ‘6’, this implies that node ‘7’ no longer exists, meaning that node ‘6’ is the new coordinator. So node ‘6’ tells all the other nodes:

Fat Clients

A fat client is a client in a client-server architecture network which typically provides rich functionality independently of the central server. Originally known as just ‘client’ or ‘thick client’, the name is contrasted to thin client, which describes a computer heavily dependant on a servers applications.

Thin Clients

A thin client is a client computer or client software in a client-server architecture network which depends primarily on the central server for processing activities, and mainly focuses on conveying input and output between the user and the remote server. In contrast, a thick or fat client does as much processing as possible and passes only data for communications and storage to the server.

Logical Clocks

Let’s say we have a logical clock, LC_i for each processor. In this case, when ever an event happens, we shall increment LC_i.

If a processor, X, sends a message to processor, Y, then processor X will also send LC_X, which is that processors logical clock.

When processor Y receives this message, then we do:

If LC_Y < (LC_X + 1):

            LC_Y = LC_X + 1

In order to update processor Y’s logical clock.

Lamport Clocks

Let’s now say that we have a processor ‘A’ and ‘B’. There are a few things we can say:

• If A precedes (happens before) B, then we can write A –> B
• If A and B are concurrent events, then sadly we can’t say anything about their ordering.

If A –> B is true, then it must also be true that LC_A < LC_B. However, this is not the case. Just because LC_A < LC_B does not mean that A –> B. Therefore, we can say that we cannot infer a casual ordering of processors just by looking at their timestamps.

Atomic Operations

An atomic operation refers to a set of operations or events that can be combined so that they appear to the rest of the system to be a single operation with only to possible outcomes: success or failure.

To read up more on this please visit:

http://en.wikipedia.org/wiki/Atomic_operation

Critical Section

In concurrent programming, a critical section is a piece of code that accesses a shared resource that must not be concurrently accessed by more than one thread of execution. [Assuming you read the link above, this can solve the problem between the 2 processes accessing the same memory location].

A critical section will usually terminate in a fixed time, and a thread, task or process will only have to wait a fixed time to enter the shared resource.

By carefully controlling which variables are modified inside and outside the critical section (usually, by accessing important state only from within), concurrent access to that state is prevented. A critical section is typically used when a multithreaded program must update multiple related variables without a separate thread making conflicting changes to that data. In a related situation, a critical section may be used to ensure a shared resource, for example a printer, can only be accessed by one process at a time.

Want more? Try here: http://en.wikipedia.org/wiki/Critical_section

So this leaves on little simple question:

Can we safely and atomically update the state on 2 different machines?

The answer? NO!

Mutual Exclusion (mutex)

Mutual exclusion algorithms are used in concurrent programming to avoid the simultaneous use of a common resource, such as a global variable, by pieces of computer code called critical sections.

For example, suppose a piece of code is altering a piece of data over several program steps, when another thread starts executing. If this second thread reads from the same piece of data which is currently being overwritten. If the second thread tries to overwrite that data, then the ensuing state will probably be unrecoverable. These shared data being accessed by critical sections of code, must therefore be protected, so that other processes which read from or write to the chunk of data are excluded from running.

A mutex is also a common name for a program object that negotiates mutual exclusion among threads, also called a lock.

For more: http://en.wikipedia.org/wiki/Mutual_exclusion

A Distributed Version of mutex

A process ‘K’ wants to enter a critical section:

• It generates a timestamp TS_K
• It then sends a request (K, TS_K) to everyone
• Waits for a reply from everyone to see if it is OK to proceed
• Process ‘K’ enters its critical section.

When another process – different from ‘K’ – receives a request:

• If this other process is in its critical section, it defers its reply
• Otherwise, it sends a reply, unless this process wishes to also try and enter the critical section
• If we want to enter the critical section, if the requestor has a smaller timestamp than us, allow them first, otherwise defer the reply.

If process ‘K’ does not get a reply from everyone, then it does not enter its critical section.

A centralized Version of mutex

Client:

• Send a request to the server for a lock on the mutex
• When a reply comes back, start the critical section
• When finished, send a message to the server to release the mutex.

The Lock Server:

• If the mutex is available, mark it as being used by that client, and send a reply. Otherwise, queue the request.
• When the mutex is released by the client, if someone else wants it, pass it to them, otherwise mark it as being available.

Two Phase Commit (2PC)

PHASE 1

            A coordinator requests a transaction, and sends a request to all participants.

For example: C1 sends a request to remove X pounds from an account and C2 sends a request to add X pounds to an account.

All participants respond as to whether they are willing and able to execute the request, and send either VOTE_COMMIT or VOTE_ABORT. Then each participant logs their vote.

PHASE 2

            The coordinator looks at all the votes, if everyone has voted to commit, then the coordinator send a GLOBAL_COMMIT to everyone, otherwise, it sends a GLOBAL_ABORT.

One receiving the decision from the coordinator, all participants records the decision locally.

If the decision is ABORT, then all participants ROLL BACK to their previous safe states, otherwise, they continue on, completing the transaction.

The Bully Algorithm

In the example of 2PC described above, how do we decide which node is the coordinator?

For this, we use an algorithm called the Bully Algorithm. This relies on some ordering and numbering of the nodes. Once this is done, we can do this:

1. Node ‘P’ sends an ELECTION to message to all other nodes with a higher number than itself.
2. If no one responds, then node ‘P’ wins the election and becomes the coordinator.
3. If one of the higher numbered nodes answers, then this algorithm repeats, but on the higher numbered node.

What if no one responds? Then quite clearly we need a time-out.

view the Bully Algorithm notes on this blog here!: Bully Algorthim

Global Clock no more!

Until the mid 1800’s, all that existed was ‘local time’ based on the suns position. This changed by about 1 minute for every 12 miles. So clearly, a global clock back then was just nonsense.

Christian Algorithm

• ‘P’ requests the time from ‘S’
• ‘S’ then prepares a response to send the time ‘T’, and then sends this response at the last possible moment.
• ‘P’ sets its time to be ‘T + RoundTripTime/2’

Problems with this, is that this algorithm assumes that the RoundTripTime is equal for both sending and receiving, so what if it isn’t?

Berkeley Algorithm

• Choose a master node

This master node can then:

• Request the time from each process.
• It then observes the RoundTripTime and estimates the time in each process and its own time.
• Averages the clock times, ignoring any outlying values.
• Sends each process a relative update to their clock, which can be positive or negative.

A problem here, is what happens to a node that has the master set its clock backwards…

Definition of a Distributed System

            A distributed System is one in which independent, self-sufficient – often autonomous or heterogeneous – spatially-separated components must use a common interconnect to exchange information and coordinate actions, and allow the whole to appear to the user as one single coherent system.

Knowledge is POWER!

Think for a moment, what could you do with a computer? Yeah, your right – sod all really, specially if its Windows . Now, what about 100 computers? That’s right, now were getting somewhere! Now, let’s look at something better. What about an entire data centre comprising thousands of computers! Well, put simply – holy monkeys that’s a lot of computers, its… its like being in heaven !

Parallel vs. Distributed

Do you remember the parallelization schemes that a distributed system can be placed into? If not, go here: http://badgerati.wordpress.com/2009/05/06/distributed-systems-interconnects/ . If it helps, its:

• SIMD
• MISD
• MIMD

Where a distributed system is MIMD-parallel computing when the interconnect is on a network-scale.

A Parallelization Example

Say you were given a list ‘a’, which contained 60 items, along with some integer ‘b’.

Now, you and 2 other people were asked to produce another different list, ‘x’, which is identical to list ‘a’, however, every item has been incremented by 1, and the first occurrence of ‘b+3’ has been removed. How would you do it?

Let’s say that it is possible to partition the work, so all 3 people can work in parallel. This would finish the job much quicker. This idea is commonly known as the divide-and-conquer strategy. And we also can say that a parallelization pattern called master-and-workers applies here.

Master-and-Worker

Master-and-worker is when you split the work load evenly between a number of processes that will work in parallel to get the correct answer. Say for example we use the example above. Here we have 3 people and a list of 60 elements. In this case you would split up the work load so each person has a list containing 20 elements from ‘a’:

• a₁ = [0:20]
• a₂ = [21:40]
• a₃ = [41:60]

Here, we will have 3 new lists: x₁, x₂ and x₃, where each element has been incremented by 1.

Now what we do it get all 3 new ‘x’ lists, and combine them together to get the one list: ‘x’, which is every element in ‘a’, but incremented by 1.

Now all we need to do it remove the first item in the list that is equal to ‘b+3’.

This process is called Split, Spawn and Merge. First we split the list, then we spawn methods to return a new list ‘x’, then we merge all the new lists together to get the one new list ‘x’.

Map-Reduce

The map-reduce model – would you believe it – comprises of the map and reduce second-order functions.

MAP

In its simplest form, a map takes a unary function ‘f’ and a collection [c₁, ···, c_n], and then returns a collection:

• [f(c₁), ….., f(c_n)]

as you can see, the map will apply the required function onto each element of a given collection. Here’s an example:

• Say we have a collection ‘A’
  >>> A = [1, 2, 3, 4, 5]   

   

• Then we have a function, say: incr(x), which will return the value ‘x+1’
  >>>def incr(x):
              return x+1
• Now we declare a new collection: ‘B’ = map(incr, A)
  >>>B = map(incr, A)

• Note how the first argument is the function incr, and the second is the collection ‘A’.

• Once this is done, the map will apply each element of ‘A’ to the function. And when we print out the new collection ‘B’ we get:
   [2, 3, 4, 5, 6]

REDUCE

In its simplest form, reduce takes a binary, associative function, Θ (with an optional, initial value ‘i’) and a collection [c₁, …, c_n], and then returns a new value:

i Θ c₁ Θ … Θ c_n

So reduce will apply the function to the elements of the given collection. Lets have an example:

• Lets use the collection ‘B’ from the map above:
  >>> B = map(incr, A)   

   

• Then we define a binary associative function, that takes to numbers for arguments:
  >>>def prodDup(x,y):
              return (x*2)*(y*2)   

   

• Now we apply the function to the collection ‘B’ using reduce:
  >>>reduce(prodDup, B)   

   

• When printed, we obtain the value 184,320.

Map-Reduce Programming Model

Users implement the following interfaces, which are 2 functions known as the mapper and the reducer.

• Map (in_key, in_value) ->
              (out_key, intermediate_value) list
• Reduce (out_key, intermediate_value list) ->
              out_value list

This infrastructure takes care of splitting, spawning and merging, as well as fault-tolerance and load balancing.

Definition of a Distributed System

            A distributed System is one in which independent, self-sufficient – often autonomous or heterogeneous – spatially-separated components must use a common interconnect to exchange information and coordinate actions, and allow the whole to appear to the user as one single coherent system.

Protocols

            In order for separate and independent processes to make sense of the communication that they are engaging in, then they must follow a set of rules (protocols) of engagement and exchange.

            A protocol is a set of rules for an InterProcess Communication (IPC). This protocol stipulates the precise sequence of events that must be enacted by the communicating processes for them to engage in communication and exchange information.

            Some communication events are unicast, that is, from one single process to another single process. So a form of one-to-one communication. And then, some events are multicast – from one single process to many other processes, so a form of one-to-many communication.

API Calling

            A protocol uses the API calls from IPC that were discussed in the Syncing and Blocking SEND and RECEIVE Processes page. These API calls were:

• SEND (receiver process) (data to be sent)
• RECEIVE – the argument for this operation is the location of where the data is to be placed, and possibly the sender process.

• ACCEPT  –  a receiver process calls this to specify its readiness to engage in communication
• CONNECT  –  the sender process calls this to initiate the engagement
• DISCONNECT — either process must use this operation to clearly disengage from communication.

In this case, the Web Server is the SENDer and the Web Browser is the RECIEVEr.

Synchronization

            REMEMBER, THERE IS NO GLOBAL CLOCK.

            Say for example, we have a timestamp t for the Web Server. In reality, this time will be later in real time, than the timestamp t’ for the Web Browser, even though it is obvious that t’ < t.

            If this is so, then the Web Browsers SEND, will have no RECEIVE counterpart in the Web Server. Because of this, data may not be exchanged between the Web Server and Web Browser.

            There is one way however, to try and avoid this problem. That is:

• The Web Browser sends a Synchronous SEND. Ie, from the page mentioned above, a SEND that waits for acknowledgement that the required data has been placed into the buffer of the receiver process.
• The Web Server keeps on looping around RECEIVE, until a SEND has been caught from the Web Browser. Remember, a RECEIVE is always Synchronous.

Sockets

Sockets are a programming abstraction which is used to implement low-level IPC. For example, 2 processes exchange information by each process having a socket of its own. They can then read/write from/onto their sockets.

Sockets are created, and if in a client-server approach, then these sockets are prepared for sending and receiving messages. With this, we can see that sending data is just done by writing to ones socket, and receiving data is done by reading from the socket.

Client-Side Sockets

A Client-side socket is best understood as the endpoint of a conversation. For example, they are short-lived. In a web browser, a socket is created to send a request, receive the response, and then once that is done, the socket is destroyed (discarded).

In order to set up a client-side, we do as follows:

• Create a socket for a given transport (TCP) and IP (IPv4).
• Connect to a server on the port corresponding to the desired protocol. (80 for HTTP).
• When the connect returns without error, then the client-side set-up is complete

Once the socket is up, the message-exchange stage can take place:

• The client sends a request for a specific HTML page, (lets say http://badgerati.wordpress.com ) through its socket.
• Then, through the same socket, it waits for a response to come, and then processes it.

Once this is done, the socket is then discarded.

Server-Side Sockets

A server socket behaves more like a dispatcher. This is because they do not normally send or receive any data: typically, they simply listen for connections on the host and port that the socket server is bound to.

When a server socket gets one connection, it also gets a new socket in response to that event. It will then spawn a handler process, or thread, which is the one that actually uses the new socket to exchange messages with the client-connected socket.

After this, the server socket then simply goes back to listen for more connections.

To set up a server socket, we follow these steps:

• Create the server socket for a given transport (TCP) and IP (IPv4).
• Set the server socket options, such as, it this socket reusable?
• Bind the server socket to a host address (localhost) and a port (80).
• Start listening to connections on the server socket and set the maximum number that could be left waiting.

The connection handling stage is a loop, which:

• The server socket first blocks waiting connections to accept connections.
• When a connection is accepted, 2 pieces of information result: the new client-connected socket has been created and the host and the port of connecting client.
• The server process then handles the connection, for example it could spawn a dispatcher passing it the socket to be used and start it.
• The server process can then continue the loop.
• (the client-connected socket must be ordered to shutdown and close when the handler is done and returns to the parent).