Jekyll2023-06-19T14:45:58+00:00http://slonka.net/feed.xmlKrzysztof SłonkaA blog about technologyMastering the Mesh: A Comprehensive Guide to Linkerd 2 Service Mesh2023-06-08T16:00:00+00:002023-06-08T16:00:00+00:00http://slonka.net/service-mesh-with-linkerd-2<p><img src="https://cncf-branding.netlify.app/img/projects/linkerd/horizontal/color/linkerd-horizontal-color.svg" alt="Linkerd Logo" /></p>
<h1 id="mastering-the-mesh-a-comprehensive-guide-to-linkerd-2-service-mesh">Mastering the Mesh: A Comprehensive Guide to Linkerd 2 Service Mesh</h1>
<p>This course is designed to teach you everything you need to know about Linkerd 2, a Service Mesh that provides a way to control and monitor the communication between services in a microservices architecture.</p>
<h2 id="what-you-will-learn">What you will learn</h2>
<ul>
<li>What is a Service Mesh?</li>
<li>What are the benefits of using a Service Mesh?</li>
<li>How Linkerd 2 works?</li>
<li>How to install and configure Linkerd 2</li>
<li>How to use Linkerd 2 to control the communication between services
<ul>
<li><strong>Service discovery:</strong> Finds services without having to hardcode IP addresses or DNS names.</li>
<li><strong>Load balancing:</strong> Spreads traffic across all instances of a service.</li>
<li><strong>Fault injection:</strong> Simulates failures to test reliability.</li>
</ul>
</li>
<li>How to use Linkerd 2 to improve ovservability, reliability, and security of microservices applications
<ul>
<li><strong>Metrics and tracing:</strong> Monitors traffic and behavior.</li>
<li><strong>Security:</strong> Protects from attacks with features like mTLS and authentication.</li>
<li><strong>Retries:</strong> Attempts to reconnect to a service that has failed.</li>
<li><strong>Timeouts:</strong> Prevents requests from hanging indefinitely.</li>
</ul>
</li>
</ul>
<h2 id="who-is-this-course-for">Who is this course for?</h2>
<p>This course is for anyone who wants to learn about Linkerd 2. This includes:</p>
<ul>
<li>Developers who are building microservices applications</li>
<li>DevOps engineers who are responsible for deploying and managing microservices applications</li>
<li>Site reliability engineers who are responsible for monitoring and troubleshooting microservices applications</li>
</ul>
<h2 id="what-you-will-get">What you will get</h2>
<ul>
<li>Live lectures and Q&A sessions with an expert instructor</li>
<li>A comprehensive course syllabus that includes all the topics that will be covered in the course</li>
<li>A certificate of completion that you can use to demonstrate your skills and knowledge</li>
</ul>
<h2 id="enroll-today-and-start-learning-about-linkerd-2">Enroll today and start learning about Linkerd 2!</h2>
<p>Use a preferred way to book a training session:</p>
<ul>
<li><a href="https://forms.gle/q8XCnbJZeGWisAJh7">Fill out form to inquire about the training</a></li>
<li><a href="mailto:trainings@slonka.net?subject=Linkerd%202%20training%20inquiry&body=Dear%20Sir%20or%20Madam,%0A%0AThank%20you%20very%20much%20for%20your%20interest%20in%20%22Mastering%20the%20Mesh%3A%20A%20Comprehensive%20Guide%20to%20Linkerd%202%20Service%20Mesh%22.%20To%20tailor%20the%20session%20to%20your%20needs%2C%20please%20provide%20the%20following%20details%3A%0A%0APreferred%20Dates%20%28in%20order%20of%20preference%29%3A%0A%5BInsert%20date%5D%0A%5BInsert%20date%5D%0A%5BInsert%20date%5D%0A%0AMode%20%28Remote%2FIn-Person%29%3A%0A%5BInsert%20preference%5D%0AIf%20In-Person%2C%20Location%3A%20%5BInsert%20location%5D%0A%0AParticipants%0ANumber%3A%20%5BInsert%20number%5D%0A%0ASpecial%20training%20needs%2Fobjectives%3A%0A%5BInsert%20needs%20and%20objectives%5D%0A%0AContact%20Details%3A%0ANumber%3A%20%5BInsert%20phone%20number%5D%0A%0ALooking%20forward%20to%20a%20successful%20session%21%0A%0ABest%20regards%2C%0AKrzysztof%20S%C5%82onka">Send Email to inquire about the training</a></li>
</ul>
<h2 id="who-am-i">Who am I?</h2>
<p>Check out my <a href="/about">about page</a>.</p>
<h2 id="languages">Languages</h2>
<p>This course is available in both 🇬🇧English and 🇵🇱Polish.</p>slonkaHow to develop envoy in CLion with bazel plugin2019-06-12T16:00:00+00:002019-06-12T16:00:00+00:00http://slonka.net/envoy-clion-bazel<h2 id="install">Install</h2>
<p>You’ll need:</p>
<ul>
<li><a href="https://www.jetbrains.com/clion/download/#section=mac">CLion</a></li>
<li><a href="https://github.com/bazelbuild/intellij/issues/494#issuecomment-498814006">Custom build</a> of <a href="https://plugins.jetbrains.com/plugin/9554-bazel">bazel plugin</a></li>
</ul>
<h2 id="debugging-project">Debugging project</h2>
<ol>
<li>Launch CLion</li>
<li>Import <code class="language-plaintext highlighter-rouge">envoy</code> via bazel plugin</li>
<li>Select WORKSPACE file <code class="language-plaintext highlighter-rouge">envoy/WORKSPACE</code></li>
<li>Next</li>
<li>Select BUILD file <code class="language-plaintext highlighter-rouge">envoy/BUILD</code></li>
<li>Wait for project to synchronize</li>
<li>Add run configuration
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Target expression: //source/exe:envoy-static
Bazel command: run
Bazel flags: -c dbg --spawn_strategy=standalone
Executable flags: --config-path PATH_TO_ENVOY_CONFIG -l debug
</code></pre></div> </div>
<p><img src="/assets/images/2019-06-12-envoy-clion-bazel/run_configuration.png" alt="run configuration" /></p>
</li>
<li>
<p>Set symbolic breakpoints (only those work, but it’s better than nothing)
<img src="/assets/images/2019-06-12-envoy-clion-bazel/breakpoints.png" alt="symbolic breakpoints clion" /></p>
</li>
<li>Click debug</li>
<li>Debugger should stop on in the main function
<img src="/assets/images/2019-06-12-envoy-clion-bazel/debugger.png" alt="debugger" /></li>
</ol>slonkaInstallRafting for fun and profit2018-11-25T20:00:00+00:002018-11-25T20:00:00+00:00http://slonka.net/rafting-for-fun-and-profit<h2 id="intro">Intro</h2>
<p>Being a part of an organization like <a href="https://allegro.tech">allegro</a>
I can participate in something that’s called “Skylab training days” which is
a day dedicated to working on something cool and interesting, not related to current sprint.
Something similar to Google’s “20% time”.
During last training days my colleagues and I implemented a Raft consensus algorithm,
that can form a cluster, elect a leader and survive failures.
You can read the paper on Raft <a href="https://raft.github.io/raft.pdf">here</a>,
but I’ll highlight the important bits in this post.</p>
<h2 id="api">API</h2>
<p>We did not implement all of Raft’s features.
In the paper you can read that:
“Raft is a consensus algorithm for managing a replicated log” -
our implementation was limited to leader election.
The API and subset of features are described in <a href="https://github.com/pbetkier/rafting">pbetkier/rafting</a>
and the repository includes a slick front-end visualization.</p>
<p>There are 3 main endpoints:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>POST /raft/request-vote - starting a vote
POST /raft/append-entries - sending heartbeats
GET /raft/state - (dashboard only) dumping raft state
</code></pre></div></div>
<p>It uses JSON over HTTP.</p>
<p>The visualization looks like this:</p>
<p><img src="/assets/images/2018-11-25-rafting-for-fun-and-profit/rafting-dashboard.png" alt="rafting dashboard" /></p>
<figcaption class="caption">Raft dashboard visualization</figcaption>
<h2 id="raft-summary">Raft summary</h2>
<p>The way I picture the algorithm is a state machine with 3 states:</p>
<ul>
<li>follower</li>
<li>candidate</li>
<li>leader</li>
</ul>
<p>The state is separate for each node of course.</p>
<p><img src="/assets/images/2018-11-25-rafting-for-fun-and-profit/state-machine.png" alt="raft sate machine" /></p>
<figcaption class="caption">Raft state changes - from raft paper</figcaption>
<p> </p>
<p>The state changes based on input from other nodes and time passing by.
Here is the summary of rules:</p>
<ol>
<li>Every node starts in a follower state.</li>
<li>If it doesn’t receive heartbeat from a leader then it transitions into candidate
and bumps current term.</li>
<li>A candidate requests votes from it’s peers and votes for himself.</li>
<li>If a node gets majority of the votes it becomes a leader.</li>
<li>A leader must send heartbeat to other nodes to make sure they are alive
and let them know who is the leader.</li>
<li>If a node receives heartbeat with higher term it changes it’s state to follower and
updates it’s term.</li>
</ol>
<h2 id="implementation">Implementation</h2>
<p>I actually wrote the algorithm twice, first time was kind of a spaghetti flavoured hack.
I did not understand the paper well enough to write tests and I just wanted to see something on the screen.
So I started with a simple object called <code class="language-plaintext highlighter-rouge">state</code>.
I hooked up controllers to every endpoint and began mutating that object based on input.
Then I added timeouts to make followers transition into candidates.
After that I added A couple of HTTP calls to request votes and… it worked! (kind of)
I run the server in 5 <a href="https://github.com/tmux/tmux">tmux</a> panes and observed the behavior.
After a couple of hours of tinkering I eventually got it right and here is the result:</p>
<iframe width="810" height="455" src="https://www.youtube.com/embed/nRrJZjvlEbk" frameborder="0" allowfullscreen=""></iframe>
<p>Let’s see an example function from the 200-ish lines of monstrosity behind this:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">async</span> <span class="kd">function</span> <span class="nx">sendHeartBeat</span><span class="p">()</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">role</span> <span class="o">===</span> <span class="nx">roles</span><span class="p">.</span><span class="nx">leader</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">peersWithoutMe</span> <span class="o">=</span> <span class="nx">getPeersWithoutMe</span><span class="p">(</span><span class="nx">peers</span><span class="p">,</span> <span class="nx">nodeId</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">responses</span> <span class="o">=</span> <span class="k">await</span> <span class="nb">Promise</span><span class="p">.</span><span class="nx">all</span><span class="p">(</span><span class="nx">peersWithoutMe</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">p</span> <span class="o">=></span> <span class="nx">request</span><span class="p">.</span><span class="nx">post</span><span class="p">({</span>
<span class="na">url</span><span class="p">:</span> <span class="s2">`https://</span><span class="p">${</span><span class="nx">p</span><span class="p">}</span><span class="s2">/raft/append-entries`</span><span class="p">,</span>
<span class="na">json</span><span class="p">:</span> <span class="p">{</span>
<span class="na">term</span><span class="p">:</span> <span class="nx">currentTerm</span><span class="p">,</span> <span class="c1">// (1)</span>
<span class="p">},</span>
<span class="na">timeout</span><span class="p">:</span> <span class="nx">REQUEST_TIMEOUT</span><span class="p">,</span>
<span class="na">resolveWithFullResponse</span><span class="p">:</span> <span class="kc">true</span>
<span class="p">})).</span><span class="nx">map</span><span class="p">(</span><span class="nx">p</span> <span class="o">=></span> <span class="nx">p</span><span class="p">.</span><span class="k">catch</span><span class="p">(</span><span class="nx">e</span> <span class="o">=></span> <span class="nx">e</span><span class="p">)));</span>
<span class="c1">// am I still the leader? (2)</span>
<span class="kd">const</span> <span class="nx">stillLeader</span> <span class="o">=</span> <span class="nx">responses</span><span class="p">.</span><span class="nx">every</span><span class="p">(</span><span class="nx">r</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">r</span><span class="p">.</span><span class="nx">statusCode</span> <span class="o">===</span> <span class="mi">200</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">r</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">term</span> <span class="o">===</span> <span class="nx">currentTerm</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">r</span><span class="p">.</span><span class="nx">statusCode</span> <span class="o">!==</span> <span class="mi">200</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="kc">true</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="c1">// (3)</span>
<span class="k">return</span> <span class="kc">false</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">stillLeader</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">setTimeout</span><span class="p">(</span><span class="nx">sendHeartBeat</span><span class="p">,</span> <span class="nx">HEARTBEAT_INTERVAL</span><span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">nodeId</span><span class="p">,</span> <span class="dl">'</span><span class="s1">lost leadership</span><span class="dl">'</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>As you can see, there is HTTP code mixed with algorithm logic (1)
a check counting votes that for some reason checks the term (2)
and some dead code (3).</p>
<p>You can see the full code of this at
<a href="https://gist.github.com/slonka/1f02a9b403891f624dffa2a121927d1c">spaghetti-raft.js</a></p>
<p>There are definitely some unnecessary if statements, checks that don’t make sense.
Making changes in this code meant I had to re-check every path I thought about manually.
After making a change I did not know the consequences, I was clueless.</p>
<p>Now I know the problem, let’s make it better.</p>
<h2 id="tdd-to-the-rescue">TDD to the rescue</h2>
<p>A state machine like that should be fairly easy to test, right?
I’ve got a bunch of input states,
bunch of transition functions
and expected output after transitions.
It’s as easy as it can get.</p>
<p>Let’s go one by one over the list of steps I outlined in the introduction.
I started with a simple test:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">it</span><span class="p">(</span><span class="dl">'</span><span class="s1">should start with initial state</span><span class="dl">'</span><span class="p">,</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="p">});</span>
</code></pre></div></div>
<p>Easy - create a state machine,
set initial state in constructor
and check it’s state is the same as the default state defined in the paper.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">emptyState</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">currentTerm</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
<span class="na">lastHeartbeat</span><span class="p">:</span> <span class="kc">null</span><span class="p">,</span>
<span class="na">role</span><span class="p">:</span> <span class="dl">'</span><span class="s1">follower</span><span class="dl">'</span><span class="p">,</span>
<span class="na">nodeId</span><span class="p">:</span> <span class="kc">null</span><span class="p">,</span>
<span class="na">votedFor</span><span class="p">:</span> <span class="kc">null</span><span class="p">,</span>
<span class="na">votesGranted</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
<span class="p">}</span>
<span class="kd">class</span> <span class="nx">RaftStateMachine</span> <span class="p">{</span>
<span class="kd">constructor</span><span class="p">({</span> <span class="nx">nodeId</span> <span class="p">})</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">nodeId</span> <span class="o">==</span> <span class="kc">undefined</span><span class="p">)</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="dl">'</span><span class="s1">nodeId and peers must be defined, peers must be an array</span><span class="dl">'</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">this</span><span class="p">.</span><span class="nx">state</span> <span class="o">=</span> <span class="p">{</span> <span class="p">...</span><span class="nx">emptyState</span><span class="p">,</span> <span class="na">peers</span><span class="p">:</span> <span class="k">new</span> <span class="nb">Set</span><span class="p">(),</span> <span class="nx">nodeId</span> <span class="p">};</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">/// ...</span>
<span class="nx">test</span><span class="p">(</span><span class="dl">'</span><span class="s1">should start with initial state</span><span class="dl">'</span><span class="p">,</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="c1">// given</span>
<span class="kd">const</span> <span class="nx">raftStateMachine</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">LocalRaftStateMachine</span><span class="p">({</span><span class="na">nodeId</span><span class="p">:</span> <span class="dl">'</span><span class="s1">host:port</span><span class="dl">'</span><span class="p">});</span>
<span class="c1">// when nothing</span>
<span class="c1">// then</span>
<span class="nx">assert</span><span class="p">.</span><span class="nx">deepEqual</span><span class="p">(</span><span class="nx">raftStateMachine</span><span class="p">.</span><span class="nx">state</span><span class="p">,</span> <span class="nx">initialState</span><span class="p">);</span>
<span class="p">});</span>
</code></pre></div></div>
<p>Next:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">test</span><span class="p">(</span><span class="dl">'</span><span class="s1">should transition from candidate to leader when received majority of votes</span><span class="dl">'</span><span class="p">,</span> <span class="k">async</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="p">});</span>
</code></pre></div></div>
<p>This one is more complex.
We start a vote if a leader timed out,
bump current term, request votes from peers then count positive votes
and if there are more positive votes the node transitions to leader state.
The only tricky part in testing this is making sure the votes are positive,
every single time.
I’m going to use <code class="language-plaintext highlighter-rouge">testdouble</code> to mock <code class="language-plaintext highlighter-rouge">getVotes</code> and <code class="language-plaintext highlighter-rouge">sendAllHeartbeats</code> functions.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">test</span><span class="p">(</span><span class="dl">'</span><span class="s1">should transition from candidate to leader when received majority of votes</span><span class="dl">'</span><span class="p">,</span> <span class="k">async</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="c1">// given</span>
<span class="kd">const</span> <span class="nx">raftStateMachine</span> <span class="o">=</span> <span class="nx">getStateMachineWithPeers</span><span class="p">();</span>
<span class="nx">td</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="nx">raftStateMachine</span><span class="p">,</span> <span class="dl">'</span><span class="s1">getVotes</span><span class="dl">'</span><span class="p">,</span> <span class="nx">td</span><span class="p">.</span><span class="nx">when</span><span class="p">(</span><span class="nx">td</span><span class="p">.</span><span class="kd">function</span><span class="p">()()).</span><span class="nx">thenResolve</span><span class="p">([</span><span class="nx">positiveVote</span><span class="p">,</span> <span class="nx">positiveVote</span><span class="p">,</span> <span class="nx">negativeVote</span><span class="p">]));</span>
<span class="nx">td</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="nx">raftStateMachine</span><span class="p">,</span> <span class="dl">'</span><span class="s1">sendAllHeartbeats</span><span class="dl">'</span><span class="p">,</span> <span class="nx">td</span><span class="p">.</span><span class="kd">function</span><span class="p">());</span>
<span class="c1">// when</span>
<span class="k">await</span> <span class="nx">raftStateMachine</span><span class="p">.</span><span class="nx">transitionToCandidate</span><span class="p">();</span>
<span class="c1">// then</span>
<span class="nx">assert</span><span class="p">.</span><span class="nx">equal</span><span class="p">(</span><span class="nx">raftStateMachine</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">role</span><span class="p">,</span> <span class="nx">roles</span><span class="p">.</span><span class="nx">leader</span><span class="p">);</span>
<span class="nx">assert</span><span class="p">.</span><span class="nx">equal</span><span class="p">(</span><span class="nx">raftStateMachine</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">currentTerm</span><span class="p">,</span> <span class="mi">2</span><span class="p">);</span>
<span class="p">});</span>
</code></pre></div></div>
<p>And the implementation:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nx">leaderTimedOut</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">now</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Date</span><span class="p">().</span><span class="nx">getTime</span><span class="p">();</span>
<span class="kd">const</span> <span class="nx">diff</span> <span class="o">=</span> <span class="nb">Math</span><span class="p">.</span><span class="nx">abs</span><span class="p">(</span><span class="nx">now</span> <span class="o">-</span> <span class="k">this</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">lastHeartbeat</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">diff</span> <span class="o">></span> <span class="nx">config</span><span class="p">.</span><span class="nx">leaderTimeout</span>
<span class="k">return</span> <span class="nx">result</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">async</span> <span class="nx">transitionToCandidate</span><span class="p">()</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">leaderTimedOut</span><span class="p">()</span> <span class="o">&&</span> <span class="k">this</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">role</span> <span class="o">!==</span> <span class="nx">roles</span><span class="p">.</span><span class="nx">leader</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">role</span> <span class="o">=</span> <span class="nx">roles</span><span class="p">.</span><span class="nx">candidate</span><span class="p">;</span>
<span class="k">this</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">currentTerm</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">votes</span> <span class="o">=</span> <span class="k">await</span> <span class="k">this</span><span class="p">.</span><span class="nx">getVotes</span><span class="p">();</span>
<span class="kd">const</span> <span class="nx">positive</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">countPositiveVotes</span><span class="p">(</span><span class="nx">votes</span><span class="p">);</span>
<span class="nx">logger</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">'</span><span class="s1">received positive votes</span><span class="dl">'</span><span class="p">,</span> <span class="nx">positive</span><span class="p">);</span>
<span class="k">this</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">votesGranted</span> <span class="o">=</span> <span class="nx">positive</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">positive</span> <span class="o">></span> <span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">peers</span><span class="p">.</span><span class="nx">size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">))</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">transitionToLeader</span><span class="p">();</span> <span class="c1">// (!)</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">resetLeaderTimeout</span><span class="p">();</span> <span class="c1">// (!)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>I marked transition changes with (!) comment.</p>
<p>So far so good, now let’s dig into something even more complex:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">it</span><span class="p">(</span><span class="dl">'</span><span class="s1">should transition to candidate role when it did not receive a heartbeat</span><span class="dl">'</span><span class="p">,</span> <span class="k">async</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="p">});</span>
</code></pre></div></div>
<p>Hmm - this one is more tricky, we need some timeouts
and keep track of the last heartbeat.
I added a <code class="language-plaintext highlighter-rouge">setTimeout</code> coupled with <code class="language-plaintext highlighter-rouge">clearTimeout</code>,
a function that will check if leader timed out - <code class="language-plaintext highlighter-rouge">leaderTimedOut()</code>
and voila: it works…</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="kd">class</span> <span class="nx">RaftStateMachine</span> <span class="p">{</span>
<span class="kd">constructor</span><span class="p">({</span> <span class="nx">nodeId</span> <span class="p">})</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">nodeId</span> <span class="o">==</span> <span class="kc">undefined</span><span class="p">)</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="dl">'</span><span class="s1">nodeId and peers must be defined, peers must be an array</span><span class="dl">'</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">this</span><span class="p">.</span><span class="nx">state</span> <span class="o">=</span> <span class="p">{</span> <span class="p">...</span><span class="nx">emptyState</span><span class="p">,</span> <span class="na">peers</span><span class="p">:</span> <span class="k">new</span> <span class="nb">Set</span><span class="p">(),</span> <span class="nx">nodeId</span> <span class="p">};</span>
<span class="k">this</span><span class="p">.</span><span class="nx">alreadyVotedInThisTerm</span> <span class="o">=</span> <span class="kc">false</span><span class="p">;</span>
<span class="k">this</span><span class="p">.</span><span class="nx">resetLeaderTimeout</span><span class="p">();</span>
<span class="p">}</span>
<span class="c1">// ...</span>
<span class="k">async</span> <span class="nx">transitionToCandidate</span><span class="p">()</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">leaderTimedOut</span><span class="p">()</span> <span class="o">&&</span> <span class="k">this</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">role</span> <span class="o">!==</span> <span class="nx">roles</span><span class="p">.</span><span class="nx">leader</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">role</span> <span class="o">=</span> <span class="nx">roles</span><span class="p">.</span><span class="nx">candidate</span><span class="p">;</span>
<span class="k">this</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">currentTerm</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">this</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">alreadyVotedInThisTerm</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">votes</span> <span class="o">=</span> <span class="k">await</span> <span class="k">this</span><span class="p">.</span><span class="nx">getVotes</span><span class="p">();</span>
<span class="kd">const</span> <span class="nx">positive</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">countPositiveVotes</span><span class="p">(</span><span class="nx">votes</span><span class="p">);</span>
<span class="nx">logger</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">'</span><span class="s1">received positive votes</span><span class="dl">'</span><span class="p">,</span> <span class="nx">positive</span><span class="p">);</span>
<span class="k">this</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">votesGranted</span> <span class="o">=</span> <span class="nx">positive</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">positive</span> <span class="o">></span> <span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">peers</span><span class="p">.</span><span class="nx">size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">))</span> <span class="p">{</span>
<span class="nx">clearTimeout</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">leaderTimeout</span><span class="p">);</span>
<span class="k">this</span><span class="p">.</span><span class="nx">transitionToLeader</span><span class="p">();</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">resetLeaderTimeout</span><span class="p">();</span> <span class="c1">// (1)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nx">resetLeaderTimeout</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">clearTimeout</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">leaderTimeout</span><span class="p">);</span>
<span class="k">this</span><span class="p">.</span><span class="nx">leaderTimeout</span> <span class="o">=</span> <span class="nx">setTimeout</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">transitionToCandidate</span><span class="p">.</span><span class="nx">bind</span><span class="p">(</span><span class="k">this</span><span class="p">),</span> <span class="nx">config</span><span class="p">.</span><span class="nx">leaderTimeout</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">// ...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>But it’s kind of “janky”<a href="https://jankfree.org/">*</a>…
Sometimes the test completes in milliseconds,
sometimes it runs for way longer…
To understand why I had to refresh my memory about event loop
and conditions for a node process to exit.</p>
<p>Every asynchronous <em>thing</em> is going to be queued up in the event loop
to be processed at the later time.
A node process will not exit if it has anything left to do in the event loop.
I made sure that the event loop was constantly occupied
by clearing a timeout an starting a new one every time I called <code class="language-plaintext highlighter-rouge">resetLeaderTimeout()</code>.</p>
<p>The situation looks like this when <code class="language-plaintext highlighter-rouge">transitionToCandidate()</code> is called:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Stack: [transitionToCandidate()]
Event loop: []
</code></pre></div></div>
<p>And by the time it exits it’s on the event loop (1)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Stack: []
Event loop: [transitionToCandidate()]
</code></pre></div></div>
<p>Ok, so how to fix it?
There is a function that can be called on a
<code class="language-plaintext highlighter-rouge">timeout</code>,
<code class="language-plaintext highlighter-rouge">immediate</code>
and <code class="language-plaintext highlighter-rouge">subprocess</code>
called <a href="https://nodejs.org/api/timers.html#timers_timeout_unref">unref</a>
which makes “object not require the Node.js event loop to remain active”.</p>
<p>Alright, we fixed that one, but it’s not ideal.
Our tests take more time than they should and it’s because we use real timers.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> PASS state-machine/state-machine.test.js
✓ should start with initial state (4ms)
✓ should not transition to leader when can not get majority (38ms)
✓ should transition to candidate role when it did not receive a heartbeat (17ms)
✓ should send heartbeats when leader (1ms)
✓ should become follower when it gets heartbeat with higher term
✓ should become candidate when it does not get heartbeat from a leader for a long time (16ms)
✓ should start a second voting when first vote is unsuccessful (16ms)
✓ should cast a vote when not voted in this term (1ms)
✓ should not cast a vote when voted in this term
✓ should elect a leader (50ms)
starting voting
✓ should transition from candidate to leader when recived majority of votes (1ms)
✓ should not transition from candidate to leader when recived less than majority of votes
</code></pre></div></div>
<p>Sum: ~144 ms</p>
<p>What can we do about it?
<a href="https://jestjs.io/docs/en/timer-mocks">Jest</a> allows mocking timers and
we’re going to use it.
The process is quite easy,
we have recursive timers so as documentation suggests,
so we’re going to run <code class="language-plaintext highlighter-rouge">jest.runOnlyPendingTimers()</code>
to only advance the timers that are scheduled and not enter infinite loop.</p>
<p>In each test we replace calls to <code class="language-plaintext highlighter-rouge">sleep()</code> with <code class="language-plaintext highlighter-rouge">jest.runOnlyPendingTimers()</code>.
One tricky test is the last one that makes sure a leader is selected.
Simply running <code class="language-plaintext highlighter-rouge">runOnlyPendingTimers()</code> doesn’t work
because <code class="language-plaintext highlighter-rouge">getVotes()</code> uses promises and
running timers is not enough because promises are handled by microtask queue in the event loop.</p>
<p>What we can do is run all pending timers,
that sets the nodes in candidate / voter phase,
replace mocked timers with real ones so we do not advance beyond that,
lastly we advance microtasks so that vote results come back to the leader.</p>
<p>After that cycle a leader should be selected.
We can run tests and see the results:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> PASS state-machine/state-machine.test.js
✓ should start with initial state (3ms)
✓ should not transition to leader when can not get majority (1ms)
✓ should transition to candidate role when it did not receive a heartbeat (2ms)
✓ should transition from candidate to leader when recived majority of votes
✓ should not transition from candidate to leader when recived less than majority of votes (1ms)
✓ should send heartbeats when leader
✓ should become follower when it gets heartbeat with higher term
✓ should become candidate when it does not get heartbeat from a leader for a long time (1ms)
✓ should start a second voting when first vote is unsuccessful
✓ should cast a vote when not voted in this term (1ms)
✓ should not cast a vote when voted in this term
✓ should elect a leader (2ms)
</code></pre></div></div>
<p>Sum: ~11 ms</p>
<p>We improved our test time by a factor of 10 - that’s pretty nice.</p>
<p>The rest of them is pretty much rinse & repeat.
One more thing that is quite cool is that I’ve abstracted a <code class="language-plaintext highlighter-rouge">LocalRaftStateMachine</code>
and <code class="language-plaintext highlighter-rouge">RemoteRaftStateMachine</code>.
The local version keeps everything in memory and is great for testing.
Nodes interact by directly calling functions on each other.
The remote version works with a hooked up server and communicates via HTTP RPC.</p>
<h2 id="summary">Summary</h2>
<p>During this exercise I’ve learned a lot, mostly:</p>
<ul>
<li>distributed consensus is hard</li>
<li>tdd can make your life easier but it’s hard to pull off when you don’t know what you’re doing</li>
<li>making small things that do not work and won’t be deployed anywhere can still be valuable</li>
</ul>
<p>You can see the full source code <a href="https://github.com/slonka/rafting-for-fun-and-profit">in this repository</a>.</p>slonkaIntroAllegro in flames2018-11-09T18:15:00+00:002018-11-09T18:15:00+00:00http://slonka.net/allegro-in-flames<h2 id="video">Video</h2>
<iframe width="560" height="310" src="https://www.youtube.com/embed/opMbDmLvqhs" frameborder="0" allowfullscreen=""></iframe>
<h2 id="presentation">Presentation</h2>
<style>
.responsive-wrap iframe{ max-width: 100%;}
</style>
<div class="responsive-wrap">
<!-- this is the embed code provided by Google -->
<iframe src="https://docs.google.com/presentation/d/e/2PACX-1vSu8-t7rJEYi-dQaCxRckBHuFy_V8-dzpHNKae8p7DFABCVzDU8V0HeYTm_h7OGQ_8fFHzQZkOMqW6P/embed?start=false&loop=false&delayms=60000" frameborder="0" width="960" height="569" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true">
</iframe>
<!-- Google embed ends -->
</div>slonkaVideoNode DIY part 1 - Distributed tracing2018-10-19T20:00:00+00:002018-10-19T20:00:00+00:00http://slonka.net/node-diy-distributed-tracing<h2 id="intro">Intro</h2>
<p>This is a first post of the series called <strong>Node DIY</strong> - a guide that
will explain how seemingly complex tools/frameworks/ideas work under the hood
and what is at the core of them.
I will try to introduce the topic and implement a bare-bones, no dependencies solution.
It will be far from production ready, but that’s the point.</p>
<p>Today’s topic is distributed tracing.
Distributed tracing is a process of identifying the path that a request (or other forms of activity, but in this article I’ll focus on HTTP)
takes traveling through every part of your system.</p>
<p><img src="/assets/images/2018-10-19-node-diy-distributed-tracing/distributed-tracing.png" alt="request tracing" /></p>
<p>When I first tried out <a href="https://docs.newrelic.com/docs/apm/distributed-tracing/getting-started/introduction-distributed-tracing">newrelic</a>
I was curious how they were able to trace all of the interactions between my services
and all it took was one function call in the application entry point.</p>
<h2 id="communication">Communication</h2>
<p>I knew that the tracing information must be kept somewhere in the HTTP request.
There is no side channel, because the library works across closed environments (different machines, vms, containers).
So I’ve setup a small two app example with newrelic running on it.
Made a request from one service to another and checked out the output.</p>
<p><img src="/assets/images/2018-10-19-node-diy-distributed-tracing/newrelic-transaction-headers.png" alt="newrelic transaction headers" /></p>
<p>And there it is: <code class="language-plaintext highlighter-rouge">x-newrelic-id</code> and <code class="language-plaintext highlighter-rouge">x-newrelic-transaction</code>.
Ok, that means that we need some way to propagate those headers and make sure we do not mix them up.</p>
<h2 id="test-servers">Test servers</h2>
<p>To test the results of your library,
we setup two servers talking to each other.
The server 1 forwards first request to server 2 with a 3 second delay
and the second request without the delay.
Server 2 just responds with status 200 and <code class="language-plaintext highlighter-rouge">'Hello, World!'</code>.
If everything goes right, we should observe log entries like this:</p>
<ol>
<li>Incoming request 1 with header 1</li>
<li>Incoming request 2 with header 2</li>
<li>Response for request 2 with header 2</li>
<li>Response for request 1 with header 1</li>
</ol>
<p>Full code below:</p>
<p>Server 1:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">./load-tracing</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">http</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">http</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">hostname</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">127.0.0.1</span><span class="dl">'</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">port</span> <span class="o">=</span> <span class="mi">3000</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">port2</span> <span class="o">=</span> <span class="mi">3001</span><span class="p">;</span>
<span class="kd">let</span> <span class="nx">delay</span> <span class="o">=</span> <span class="mi">3</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">server</span> <span class="o">=</span> <span class="nx">http</span><span class="p">.</span><span class="nx">createServer</span><span class="p">((</span><span class="nx">req</span><span class="p">,</span> <span class="nx">res</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">'</span><span class="s1">s1: curl headers</span><span class="dl">'</span><span class="p">,</span> <span class="nx">req</span><span class="p">.</span><span class="nx">headers</span><span class="p">);</span>
<span class="nx">setTimeout</span><span class="p">(()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">http</span><span class="p">.</span><span class="nx">request</span><span class="p">({</span><span class="na">host</span><span class="p">:</span> <span class="nx">hostname</span><span class="p">,</span> <span class="na">port</span><span class="p">:</span> <span class="nx">port2</span><span class="p">},</span> <span class="p">(</span><span class="nx">response</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">'</span><span class="s1">s1: response</span><span class="dl">'</span><span class="p">,</span> <span class="nx">response</span><span class="p">.</span><span class="nx">headers</span><span class="p">);</span>
<span class="nx">res</span><span class="p">.</span><span class="nx">statusCode</span> <span class="o">=</span> <span class="mi">200</span><span class="p">;</span>
<span class="nx">res</span><span class="p">.</span><span class="nx">setHeader</span><span class="p">(</span><span class="dl">'</span><span class="s1">Content-Type</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">text/plain</span><span class="dl">'</span><span class="p">);</span>
<span class="nx">res</span><span class="p">.</span><span class="nx">end</span><span class="p">(</span><span class="dl">'</span><span class="s1">Hello, World!</span><span class="dl">'</span><span class="p">);</span>
<span class="p">}).</span><span class="nx">end</span><span class="p">();</span>
<span class="p">},</span> <span class="nx">delay</span> <span class="o">*</span> <span class="mi">1000</span><span class="p">);</span>
<span class="nx">delay</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">});</span>
<span class="nx">server</span><span class="p">.</span><span class="nx">listen</span><span class="p">(</span><span class="nx">port</span><span class="p">,</span> <span class="nx">hostname</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">`Server running at https://</span><span class="p">${</span><span class="nx">hostname</span><span class="p">}</span><span class="s2">:</span><span class="p">${</span><span class="nx">port</span><span class="p">}</span><span class="s2">/`</span><span class="p">);</span>
<span class="p">});</span>
</code></pre></div></div>
<p>Server 2:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">./load-tracing</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">http</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">http</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">hostname</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">127.0.0.1</span><span class="dl">'</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">port2</span> <span class="o">=</span> <span class="mi">3001</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">server2</span> <span class="o">=</span> <span class="nx">http</span><span class="p">.</span><span class="nx">createServer</span><span class="p">((</span><span class="nx">req</span><span class="p">,</span> <span class="nx">res</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">res</span><span class="p">.</span><span class="nx">statusCode</span> <span class="o">=</span> <span class="mi">200</span><span class="p">;</span>
<span class="nx">res</span><span class="p">.</span><span class="nx">setHeader</span><span class="p">(</span><span class="dl">'</span><span class="s1">Content-Type</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">text/plain</span><span class="dl">'</span><span class="p">);</span>
<span class="nx">res</span><span class="p">.</span><span class="nx">end</span><span class="p">(</span><span class="dl">'</span><span class="s1">Hello, World!</span><span class="dl">'</span><span class="p">);</span>
<span class="p">});</span>
<span class="nx">server2</span><span class="p">.</span><span class="nx">listen</span><span class="p">(</span><span class="nx">port2</span><span class="p">,</span> <span class="nx">hostname</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">`Server running at https://</span><span class="p">${</span><span class="nx">hostname</span><span class="p">}</span><span class="s2">:</span><span class="p">${</span><span class="nx">port2</span><span class="p">}</span><span class="s2">/`</span><span class="p">);</span>
<span class="p">});</span>
</code></pre></div></div>
<h2 id="part-1---shimming">Part 1 - Shimming</h2>
<p>Node.js has a method that is called every time there is an event associated with a server (for example a request comes in)
and there is another method that is called when we create an outgoing connection.
They are <code class="language-plaintext highlighter-rouge">http.Server.emit</code> and <code class="language-plaintext highlighter-rouge">http.request</code> respectively.
Our goal is to change them so that we can add our own custom headers there.</p>
<p>JavaScript being a language that is easily extensible,
allows replacing built-in functions by simply assigning a new function to them.
This is quite awesome but you have to be extremely careful not to screw things up.</p>
<p>The act of making such modifications is called <a href="https://en.wikipedia.org/wiki/Shim_(computing)">shimming</a>.
Ok so let’s try and implement a basic shimmer:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">wrap</span><span class="p">(</span><span class="nx">nodule</span><span class="p">,</span> <span class="nx">name</span><span class="p">,</span> <span class="nx">wrapper</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">original</span> <span class="o">=</span> <span class="nx">nodule</span><span class="p">[</span><span class="nx">name</span><span class="p">];</span>
<span class="kd">const</span> <span class="nx">wrapped</span> <span class="o">=</span> <span class="nx">wrapper</span><span class="p">(</span><span class="nx">original</span><span class="p">);</span>
<span class="nx">nodule</span><span class="p">[</span><span class="nx">name</span><span class="p">]</span> <span class="o">=</span> <span class="nx">wrapped</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Just 3 lines, we get the original function, wrap it in a wrapper, and replace it.
That simple. There is one requirement for the wrapper function,
it has to be a function that takes the original function as an argument
and returns a function which takes the arguments of the original function.</p>
<h2 id="part-2---unique-identifiers">Part 2 - unique identifiers</h2>
<p>We need some way to distinguish between requests - this is a short implementation of uuid v4
taken from <a href="https://stackoverflow.com/a/2117523">stackoverflow</a> and modified to be easier to read:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">crypto</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">crypto</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">pattern</span> <span class="o">=</span> <span class="s2">`</span><span class="p">${</span><span class="mi">1</span><span class="nx">e7</span><span class="p">}</span><span class="s2">-</span><span class="p">${</span><span class="mi">1</span><span class="nx">e3</span><span class="p">}</span><span class="s2">-</span><span class="p">${</span><span class="mi">4</span><span class="nx">e3</span><span class="p">}</span><span class="s2">-</span><span class="p">${</span><span class="mi">8</span><span class="nx">e3</span><span class="p">}</span><span class="s2">-</span><span class="p">${</span><span class="mi">1</span><span class="nx">e11</span><span class="p">}</span><span class="s2">`</span><span class="p">;</span>
<span class="kd">function</span> <span class="nx">uuidv4</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">pattern</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span>
<span class="sr">/</span><span class="se">[</span><span class="sr">018</span><span class="se">]</span><span class="sr">/g</span><span class="p">,</span>
<span class="nx">c</span> <span class="o">=></span> <span class="p">(</span><span class="nx">c</span> <span class="o">^</span> <span class="nx">crypto</span><span class="p">.</span><span class="nx">randomBytes</span><span class="p">(</span><span class="mi">1</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span> <span class="o">&</span> <span class="mi">15</span> <span class="o">>></span> <span class="nx">c</span> <span class="o">/</span> <span class="mi">4</span><span class="p">).</span><span class="nx">toString</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span>
<span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Nothing fancy, just random bits.</p>
<h2 id="part-3---naive-approach">Part 3 - naive approach</h2>
<p>So far we know how to inject our custom headers with some unique identifier,
but how do we know keep track of the headers when we have multiple requests coming through?</p>
<p>A naive implementation might look something like this:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span> <span class="nx">uuid</span><span class="p">;</span>
<span class="nx">wrap</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">Server</span><span class="p">.</span><span class="nx">prototype</span><span class="p">,</span> <span class="dl">'</span><span class="s1">emit</span><span class="dl">'</span><span class="p">,</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">original</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">event</span><span class="p">,</span> <span class="nx">incomingMessage</span><span class="p">,</span> <span class="nx">serverResponse</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">let</span> <span class="nx">returned</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">event</span> <span class="o">===</span> <span class="dl">'</span><span class="s1">request</span><span class="dl">'</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">incomingMessage</span><span class="p">.</span><span class="nx">headers</span><span class="p">[</span><span class="nx">PROPAGATE_HEADER_NAME</span><span class="p">])</span> <span class="p">{</span>
<span class="nx">uuid</span> <span class="o">=</span> <span class="nx">incomingMessage</span><span class="p">.</span><span class="nx">headers</span><span class="p">[</span><span class="nx">PROPAGATE_HEADER_NAME</span><span class="p">];</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">uuid</span> <span class="o">=</span> <span class="nx">uuidv4</span><span class="p">();</span>
<span class="p">}</span>
<span class="nx">serverResponse</span><span class="p">.</span><span class="nx">setHeader</span><span class="p">(</span><span class="nx">PROPAGATE_HEADER_NAME</span><span class="p">,</span> <span class="nx">uuid</span><span class="p">);</span>
<span class="nx">returned</span> <span class="o">=</span> <span class="nx">original</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="nx">arguments</span><span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">returned</span> <span class="o">=</span> <span class="nx">original</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="nx">arguments</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nx">returned</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">});</span>
<span class="nx">wrap</span><span class="p">(</span><span class="nx">http</span><span class="p">,</span> <span class="dl">'</span><span class="s1">request</span><span class="dl">'</span><span class="p">,</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">original</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">options</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">options</span><span class="p">.</span><span class="nx">headers</span> <span class="o">=</span> <span class="nx">options</span><span class="p">.</span><span class="nx">headers</span> <span class="o">||</span> <span class="p">{};</span>
<span class="nx">options</span><span class="p">.</span><span class="nx">headers</span><span class="p">[</span><span class="nx">PROPAGATE_HEADER_NAME</span><span class="p">]</span> <span class="o">=</span> <span class="nx">uuid</span><span class="p">;</span>
<span class="k">return</span> <span class="nx">original</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="nx">arguments</span><span class="p">);</span>
<span class="p">};</span>
<span class="p">});</span>
</code></pre></div></div>
<p>When we run this code in our test servers it outputs:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>s1: curl headers { host: 'localhost:3000',
'user-agent': 'curl/7.54.0',
accept: '*/*',
'x-correlation-id': '1' }
s1: curl headers { host: 'localhost:3000',
'user-agent': 'curl/7.54.0',
accept: '*/*',
'x-correlation-id': '2' }
s1: response { 'x-correlation-id': '2',
'content-type': 'text/plain',
date: 'Tue, 30 Oct 2018 22:18:56 GMT',
connection: 'close',
'content-length': '14' }
Hello, World!
s1: response { 'x-correlation-id': '2',
'content-type': 'text/plain',
date: 'Tue, 30 Oct 2018 22:18:58 GMT',
connection: 'close',
'content-length': '14' }
Hello, World!
</code></pre></div></div>
<p>Hmm… The second response has the wrong header.</p>
<p>What’s wrong with this? We wrap the <code class="language-plaintext highlighter-rouge">emit</code> function, assign a uuid or propagate the header.
When we create an outgoing connection we assign the uuid we created before to the new request.
And there is the bug, if we create a request before the response from the previous one comes in
we overwrite the value.</p>
<p>How can we solve this?</p>
<p>We need some mechanism that is aware of the execution context.</p>
<h2 id="part-4---async-hooks">Part 4 - async hooks</h2>
<p>In JavaScript whenever you invoke an asynchronous action you create a resource with an async id.
Those invocations are tied together by a parent - child relationship via triggered id.</p>
<p>A simple example will show how this works:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">asyncHooks</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">async_hooks</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">fs</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">fs</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">util</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">util</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">function</span> <span class="nx">debug</span><span class="p">(...</span><span class="nx">args</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">fs</span><span class="p">.</span><span class="nx">writeSync</span><span class="p">(</span><span class="nx">process</span><span class="p">.</span><span class="nx">stdout</span><span class="p">.</span><span class="nx">fd</span><span class="p">,</span> <span class="s2">`</span><span class="p">${</span><span class="nx">util</span><span class="p">.</span><span class="nx">format</span><span class="p">(...</span><span class="nx">args</span><span class="p">)}</span><span class="s2">`</span><span class="p">);</span>
<span class="p">}</span>
<span class="nx">asyncHooks</span><span class="p">.</span><span class="nx">createHook</span><span class="p">({</span>
<span class="nx">init</span><span class="p">(</span><span class="nx">asyncId</span><span class="p">,</span> <span class="nx">type</span><span class="p">,</span> <span class="nx">triggerAsyncId</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">debug</span><span class="p">(</span><span class="s2">`Init </span><span class="p">${</span><span class="nx">type</span><span class="p">}</span><span class="s2"> resource: asyncId: </span><span class="p">${</span><span class="nx">asyncId</span><span class="p">}</span><span class="s2"> trigger: </span><span class="p">${</span><span class="nx">triggerAsyncId</span><span class="p">}</span><span class="s2">`</span><span class="p">);</span>
<span class="p">},</span>
<span class="nx">destroy</span><span class="p">(</span><span class="nx">asyncId</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">eid</span> <span class="o">=</span> <span class="nx">asyncHooks</span><span class="p">.</span><span class="nx">executionAsyncId</span><span class="p">();</span>
<span class="nx">debug</span><span class="p">(</span><span class="s2">`Destroy resource: execution: </span><span class="p">${</span><span class="nx">eid</span><span class="p">}</span><span class="s2"> asyncId: </span><span class="p">${</span><span class="nx">asyncId</span><span class="p">}</span><span class="s2">`</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}).</span><span class="nx">enable</span><span class="p">();</span>
<span class="nx">debug</span><span class="p">(</span><span class="s2">`in top level: asyncId: </span><span class="p">${</span><span class="nx">asyncHooks</span><span class="p">.</span><span class="nx">executionAsyncId</span><span class="p">()}</span><span class="s2"> trigger: </span><span class="p">${</span><span class="nx">asyncHooks</span><span class="p">.</span><span class="nx">triggerAsyncId</span><span class="p">()}</span><span class="s2">`</span><span class="p">);</span>
<span class="nx">setTimeout</span><span class="p">(()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">debug</span><span class="p">(</span><span class="s2">`in timeout: asyncId: </span><span class="p">${</span><span class="nx">asyncHooks</span><span class="p">.</span><span class="nx">executionAsyncId</span><span class="p">()}</span><span class="s2"> trigger: </span><span class="p">${</span><span class="nx">asyncHooks</span><span class="p">.</span><span class="nx">triggerAsyncId</span><span class="p">()}</span><span class="s2">`</span><span class="p">);</span>
<span class="p">},</span> <span class="mi">1000</span><span class="p">);</span>
</code></pre></div></div>
<p>This outputs:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>in top level: asyncId: 1 trigger: 0
Init Timeout resource: asyncId: 5 trigger: 1
Init TIMERWRAP resource: asyncId: 6 trigger: 1
in timeout: asyncId: 5 trigger: 1
Destroy resource: execution: 0 asyncId: 5
Destroy resource: execution: 0 asyncId: 6
</code></pre></div></div>
<p>We see that we can track each resource creation
and then at the moment of invocation check the id using <code class="language-plaintext highlighter-rouge">executionAsyncId</code>.
Using these methods we can create an execution context
and store arbitrary information there.
Here is a full implementation
(the code is based on <a href="https://github.com/guyguyon/node-request-context/blob/master/namespace.js">guyguyon</a> but modified and simplified),
let’s go through this:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">asyncHooks</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">async_hooks</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">function</span> <span class="nx">getNamespace</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">context</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Map</span><span class="p">();</span>
<span class="kd">const</span> <span class="nx">run</span> <span class="o">=</span> <span class="p">(</span><span class="nx">fn</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">eid</span> <span class="o">=</span> <span class="nx">asyncHooks</span><span class="p">.</span><span class="nx">executionAsyncId</span><span class="p">();</span>
<span class="nx">context</span><span class="p">.</span><span class="kd">set</span><span class="p">(</span><span class="nx">eid</span><span class="p">,</span> <span class="k">new</span> <span class="nb">Map</span><span class="p">());</span>
<span class="nx">fn</span><span class="p">();</span>
<span class="p">}</span>
<span class="kd">const</span> <span class="kd">set</span> <span class="o">=</span> <span class="p">(</span><span class="nx">key</span><span class="p">,</span> <span class="nx">val</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">eid</span> <span class="o">=</span> <span class="nx">asyncHooks</span><span class="p">.</span><span class="nx">executionAsyncId</span><span class="p">();</span>
<span class="nx">context</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="nx">eid</span><span class="p">).</span><span class="kd">set</span><span class="p">(</span><span class="nx">key</span><span class="p">,</span> <span class="nx">val</span><span class="p">);</span>
<span class="p">}</span>
<span class="kd">const</span> <span class="kd">get</span> <span class="o">=</span> <span class="p">(</span><span class="nx">key</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">eid</span> <span class="o">=</span> <span class="nx">asyncHooks</span><span class="p">.</span><span class="nx">executionAsyncId</span><span class="p">();</span>
<span class="k">return</span> <span class="nx">context</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="nx">eid</span><span class="p">).</span><span class="kd">get</span><span class="p">(</span><span class="nx">key</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="p">{</span>
<span class="nx">run</span><span class="p">,</span>
<span class="kd">set</span><span class="p">,</span>
<span class="kd">get</span><span class="p">,</span>
<span class="nx">context</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">createHooks</span><span class="p">(</span><span class="nx">namespace</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">function</span> <span class="nx">init</span><span class="p">(</span><span class="nx">asyncId</span><span class="p">,</span> <span class="nx">type</span><span class="p">,</span> <span class="nx">triggerId</span><span class="p">,</span> <span class="nx">resource</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">namespace</span><span class="p">.</span><span class="nx">context</span><span class="p">.</span><span class="nx">has</span><span class="p">(</span><span class="nx">triggerId</span><span class="p">))</span> <span class="p">{</span>
<span class="nx">namespace</span><span class="p">.</span><span class="nx">context</span><span class="p">.</span><span class="kd">set</span><span class="p">(</span><span class="nx">asyncId</span><span class="p">,</span> <span class="nx">namespace</span><span class="p">.</span><span class="nx">context</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="nx">triggerId</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">destroy</span><span class="p">(</span><span class="nx">asyncId</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">namespace</span><span class="p">.</span><span class="nx">context</span><span class="p">.</span><span class="k">delete</span><span class="p">(</span><span class="nx">asyncId</span><span class="p">);</span>
<span class="p">}</span>
<span class="kd">const</span> <span class="nx">hook</span> <span class="o">=</span> <span class="nx">asyncHooks</span><span class="p">.</span><span class="nx">createHook</span><span class="p">({</span>
<span class="nx">init</span><span class="p">,</span>
<span class="nx">destroy</span>
<span class="p">});</span>
<span class="nx">hook</span><span class="p">.</span><span class="nx">enable</span><span class="p">();</span>
<span class="p">}</span>
<span class="kd">const</span> <span class="nx">namespace</span> <span class="o">=</span> <span class="nx">getNamespace</span><span class="p">();</span>
<span class="nx">createHooks</span><span class="p">(</span><span class="nx">namespace</span><span class="p">);</span>
<span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span>
<span class="nx">namespace</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We start with a <code class="language-plaintext highlighter-rouge">getNamespace</code> function. It creates a new <code class="language-plaintext highlighter-rouge">context</code> which is
something we will put data associated with an execution <em>context</em> in.
Then we have three functions:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">run</code> - gets the async id, and creates a new <code class="language-plaintext highlighter-rouge">Map</code> to store information</li>
<li><code class="language-plaintext highlighter-rouge">set</code> - sets a key/value in a context</li>
<li><code class="language-plaintext highlighter-rouge">get</code> - gets a value from a context given a key</li>
</ul>
<p>Then we create an async hook,
in <code class="language-plaintext highlighter-rouge">init</code> function we copy the parent’s value from the context to a child
so that it can be accessed when it’s needed (further down the async stack).
In <code class="language-plaintext highlighter-rouge">destroy</code> function we simply delete the value from the context.</p>
<p>Now everything is ready to be connected together.</p>
<h2 id="part-5---tying-everything-together">Part 5 - tying everything together</h2>
<p>Ok, now we can get back to the tracing code.</p>
<p>All we need to do is move the uuid to wrapped function code,
<code class="language-plaintext highlighter-rouge">set</code> it in the context
and <code class="language-plaintext highlighter-rouge">get</code> the uuid from the context when wrapping <code class="language-plaintext highlighter-rouge">http.request</code>.</p>
<p>Here is the full code:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">http</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">http</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="p">{</span><span class="nx">uuidv4</span><span class="p">}</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">./uuid</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="p">{</span><span class="nx">wrap</span><span class="p">}</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">./shimmer</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="p">{</span><span class="nx">namespace</span><span class="p">}</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">./namespace</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">PROPAGATE_HEADER_NAME</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">x-correlation-id</span><span class="dl">'</span><span class="p">;</span>
<span class="nx">wrap</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">Server</span><span class="p">.</span><span class="nx">prototype</span><span class="p">,</span> <span class="dl">'</span><span class="s1">emit</span><span class="dl">'</span><span class="p">,</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">original</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">event</span><span class="p">,</span> <span class="nx">incomingMessage</span><span class="p">,</span> <span class="nx">serverResponse</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">let</span> <span class="nx">returned</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">event</span> <span class="o">===</span> <span class="dl">'</span><span class="s1">request</span><span class="dl">'</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">let</span> <span class="nx">uuid</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">incomingMessage</span><span class="p">.</span><span class="nx">headers</span><span class="p">[</span><span class="nx">PROPAGATE_HEADER_NAME</span><span class="p">])</span> <span class="p">{</span>
<span class="nx">uuid</span> <span class="o">=</span> <span class="nx">incomingMessage</span><span class="p">.</span><span class="nx">headers</span><span class="p">[</span><span class="nx">PROPAGATE_HEADER_NAME</span><span class="p">];</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">uuid</span> <span class="o">=</span> <span class="nx">uuidv4</span><span class="p">();</span>
<span class="p">}</span>
<span class="nx">serverResponse</span><span class="p">.</span><span class="nx">setHeader</span><span class="p">(</span><span class="nx">PROPAGATE_HEADER_NAME</span><span class="p">,</span> <span class="nx">uuid</span><span class="p">);</span>
<span class="nx">namespace</span><span class="p">.</span><span class="nx">run</span><span class="p">(()</span> <span class="o">=></span> <span class="p">{</span> <span class="c1">// here we set the uuid in a namespace</span>
<span class="nx">namespace</span><span class="p">.</span><span class="kd">set</span><span class="p">(</span><span class="dl">'</span><span class="s1">tid</span><span class="dl">'</span><span class="p">,</span> <span class="nx">uuid</span><span class="p">);</span>
<span class="nx">returned</span> <span class="o">=</span> <span class="nx">original</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="nx">arguments</span><span class="p">);</span>
<span class="p">});</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">returned</span> <span class="o">=</span> <span class="nx">original</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="nx">arguments</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nx">returned</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">});</span>
<span class="nx">wrap</span><span class="p">(</span><span class="nx">http</span><span class="p">,</span> <span class="dl">'</span><span class="s1">request</span><span class="dl">'</span><span class="p">,</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">original</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">options</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">options</span><span class="p">.</span><span class="nx">headers</span> <span class="o">=</span> <span class="nx">options</span><span class="p">.</span><span class="nx">headers</span> <span class="o">||</span> <span class="p">{};</span>
<span class="nx">options</span><span class="p">.</span><span class="nx">headers</span><span class="p">[</span><span class="nx">PROPAGATE_HEADER_NAME</span><span class="p">]</span> <span class="o">=</span> <span class="nx">namespace</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">'</span><span class="s1">tid</span><span class="dl">'</span><span class="p">);</span> <span class="c1">// here we read from it</span>
<span class="k">return</span> <span class="nx">original</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="nx">arguments</span><span class="p">);</span>
<span class="p">};</span>
<span class="p">});</span>
</code></pre></div></div>
<p>Running this in our test bench yields:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>s1: curl headers { host: 'localhost:3000',
'user-agent': 'curl/7.54.0',
accept: '*/*',
'x-correlation-id': '1' }
s1: curl headers { host: 'localhost:3000',
'user-agent': 'curl/7.54.0',
accept: '*/*',
'x-correlation-id': '2' }
s1: response { 'x-correlation-id': '2',
'content-type': 'text/plain',
date: 'Tue, 30 Oct 2018 22:25:58 GMT',
connection: 'close',
'content-length': '14' }
Hello, World!
s1: response { 'x-correlation-id': *'1'*,
'content-type': 'text/plain',
date: 'Tue, 30 Oct 2018 22:26:00 GMT',
connection: 'close',
'content-length': '14' }
Hello, World!
</code></pre></div></div>
<p>Which is exactly what we wanted.</p>
<h1 id="summary">Summary</h1>
<p>We have successfully created a POC distributed tracing library.
Of course there are more aspects to distributed tracing like: sampling, spans, collecting data, visualization,
but this is still very impressive that we were able to do this in approximately 100 lines of code.</p>
<p>You can find complete source code <a href="https://github.com/slonka/node-diy-distributed-tracing">on my github</a>.</p>slonkaIntro