deephacks

Watching the commercial software industry frenetically trying to make process work occasionally feels similar to watching tragicomedy. This constant struggle of trying to force their feet into these tailor-made development processes left and right: agile, lean, scrum, kanban, even waterfall, whatever, just because some manager or “tech-lead” read a blog post. The Emperor’s New Groove, eh?

It can be a really shocking experience to observe this rather bizarre circus sometimes.

But what is even more astounding is that these corporations often seem ignorant towards open source cultures and ideals. Do they know they exist? Are they ignorant? Or is decentralization and meritocracy so scary (even threatening) to leaders of the organization? Maybe. Who knows. Go figure.

But in the context of corporations, I cannot help thinking about people like Jim Whitehurst, President and CEO of Red Hat. This man seems to be a humble and smart guy, wanting the best for his people and company. What has Jim seen that so many others have not?

Well, if you work at a company that struggle with process without getting any real work done, you first have to realize that “Culture is King”. You will never get process right without culture, ideals and beliefs. One important difference between culture and process is that culture is never forced. Culture is more like a garden or family. Treat it carefully, give it freedom and nourishment to grow and eventually (with good seeds of course) it will flourish and process emerge in a strong and natural way.

I’m not saying that open source is the answer per se; not everything must be open source, but open source culture accumulates so much timeless experience, healthy principles and working ethics. On the contrary, corporations often lose invaluable information along with people leaving them and they constantly struggle to re-build expertise and educate their workforce.

Anyway, here is a short reading list (in no particular order) for those of who want to know more about open source culture, community collaboration and (in opinion) the most appealing and practical way of doing software development.

(Comments on each reading is not mine, but from its author(s) or other source(s))

Open Advice – Misc
Open Advice is a knowledge collection from a wide variety of Free Software projects. It answers the question what 42 prominent contributors would have liked to know when they started so you can get a head-start no matter how and where you contribute.

The Open Source Way – Red Hat Community Architecture team
Guide for helping people to understand how to and how not to engage with community over projects such as software, content, marketing, art, infrastructure, standards, and so forth. It contains knowledge distilled from years of Red Hat experience.

Open Source Community Values – Jeff Cohen
Welcome to Our Community. Here Are the Ground Rules.

The Art of Community: Building the New Age of Participation – Jono Bacon
Will help you develop the broad range of talents you need to recruit members to your community, motivate and manage them, and help them become active participants.

Producing Open Source Software – Karl Fogel
A book about the human side of open source development. It describes how successful projects operate, the expectations of users and developers, and the culture of free software.

Open Sources: Voices from the Open Source Revolution – Misc
Leaders of Open Source come together for the first time to discuss the new vision of the software industry they have created. The essays in this volume offer insight into how the Open Source movement works, why it succeeds, and where it is going.

Debian Constitution – The Debian Project
This document describes the organisational structure for formal decision-making in the Debian Project.

The Cathedral and the Bazaar – Eric Steven Raymond
Surprising theories about software engineering suggested by the history of Linux.

The Art of Unix Programming – Eric Steven Raymond
This book has a lot of knowledge in it, but it is mainly about expertise. It is going to try to teach you the things about Unix development that Unix experts know, but aren’t aware that they know.

How To Ask Questions The Smart Way – Eric Steven Raymond
In the world of hackers, the kind of answers you get to your technical questions depends as much on the way you ask the questions as on the difficulty of developing the answer. This guide will teach you how to ask questions in a way more likely to get you a satisfactory answer.

How the ASF works – Apache Software Foundation
Will give you everything you always wanted to know about ASF but were afraid to ask.

Apache Subversion Community Guide – Subversion Community
Subversion community participation guidelines.

Python Community Diversity Statement – Python Community
The Python Software Foundation and the global Python community welcome and encourage participation by everyone. Our community is based on mutual respect, tolerance, and encouragement, and we are working to help each other live up to these principles.

Eclipse Development Process – The Eclipse Foundation
This document describes the Development Process for the Eclipse Foundation.

Ubuntu Code of Conduct – Ubuntu Community
This Code of Conduct covers our behaviour as members of the Ubuntu Community, in any forum, mailing list, wiki, web site, IRC channel, install-fest, public meeting or private correspondence.

Mozilla Code of Conduct (draft) – Mozilla Foundation
This Code of Conduct covers our behaviour as members of the Mozilla Community, in any forum, mailing list, wiki, web site, IRC channel, bug, event, public meeting or private correspondence.

Communities of practice – Etienne Wenger
This brief and general introduction examines what communities of practice are and why researchers and practitioners in so many different contexts find them useful as an approach to knowing and learning.

Teaching Open Source – Misc
This is a neutral collaboration point for professors, institutions, communities, and companies to come together and make the teaching of Open Source a global success.

If you believe that you know anything of Configuration Management then you should know who Mike Burgess is. If not – then my guess is that you don’t have a clue about DevOps and Cloud Computing either.

Mike Burgess has studied the field of Configuration Management for last 20 years. He is the father of CFEngine, a framework for system automation that also heavily inspired others such as Puppet and Chef. I have studied Mikes research for the past year and it is incredible interesting – triggering many eureka moments.

Anyway, I just saw a this very interesting talk by Mike from the Velocity 2011 conference held this summer.

In relation to this – I can also recommend reading Rethinking Application
Delivery in the Age of Complexity and Why Scripting Is Evil by Erik Troan, inventor of RPM and rPath.

Highly recommended.

I just went to a javaforum to celebrate the release of Java 7 with some friends, happy to see that some of the swedish JRockit core team was there to give their view on Java from an Oracle perspective.

The event was immediately full-booked on the day of announcement. And the interest was so high that the promoters quickly organized one more opportunity to celebrate in the coming two weeks, which was also immediately full-booked! Considering the bad economy and outsourcing insanity going on in the industry it is really nice to see Java is really alive and kickin’ it in Sweden

Another great surprise was also revealed at the event, Uncle Bob was present to give a talk on Clean Code. Seeing living legends in person does not happen very often in Sweden unfortunately. I was stoked!

Among others, Robert talked about the importance of short methods. Hell, i want my methods to be short. I usually think 10 lines is reasonable.. but the man insisted on 3!

This made me wonder… what does he think about the Builder pattern where multiple method calls are chained together in a one-liner to create a fluent interface? Would that count as one line even though multiple method calls would be nested within that line of code?

Just for the record: im not talking about a transitive coupling train-wreck here.

Unfortunately I could not manage to ask my question since the QnA was very short, with only two questions answered. But I guess he would approve it as one line, as long as the syntax would be understandable.

Anyways, it was a really good and energizing presentation from Uncle Bob *and* Oracle, not to forget the free beer

I want code to be simple-n-short, on-point and easy to read. Unnecessary complexity distract and obscure understanding of what is really going on and can be a real killer for productivity.

You know, tangled for-loops and indexes to track, if/else and switch cases, null/validation checks, converting/copying/deleting/sorting collections, exception handling … the list goes on along with ever-increasing line numbers and maintenance burden.

An excellent quote by Tony Hoare comes to mind.

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.

In other words: The Devil is in the details.

Apache Commons have some of most wonderful libraries complementing the JDK APIs, but this post is not about Commons. It is about Google Guava which is similar to Commons in many regards. It provide a single library for commonly used day-to-day tasks, such as collection handling, string manipulation, concurrency, IO, primitives, exceptions etc.

There is so much nice stuff in Guava and I wont have time to go through the complete library, but here at least some examples of what it can do.

Objects
Objects makes it easy to implement hashcode/equals without cluttering your classes too much (Eclipse auto-generation tends to be a bit verbose for my taste).

Classes that implement toString are really pleasant to use when doing debugging and logging, but can be a real pain implement. Objects.toStringHelper makes this really easy and also help maintaining a consistent format for printing objects.

public class Item {
  private String id;
  private String name;
  public Item(String id, String name) {
    this.id = id;
    this.id = name;
  }
  public String getId() { return id;}
  public String getName() { return name; }
 
  @Override
  public int hashCode() {
    return Objects.hashCode(getId(), getName());
  }
 
  @Override
  public String toString() {
    return Objects.toStringHelper(this).add("id", getId()).add("name", getName()).toString();
  }
 
  @Override
  public boolean equals(Object o) {
    if (!(o instanceof Item)) {
      return false;
    }
    Item other = (Item) o;
    return Objects.equal(getId(), other.getId(); 
    Objects.equal(getName(), other.getName());
  }
}

Printing this class outputs something like this.

Item{id=1, name=Book}

Throwables
Wrapping the original exception object is not always appropriate, because it can cause ClassNotFoundException in the client code if communication occur between unrelated class loaders or if they are serialized on the wire. Throwables can decouple this dependency, still allowing remote clients to see the stack trace by converting it to a string.

try {
  // throws implementation specific exception
} catch (InternalException e) {
  throw new ApiException("reason", Throwables.getStackTraceAsString(e));
}

Iterables
Concatenating two separate collections and performing operations on the result can cause a quite a lot of clutter. Iterables to the rescue. Take a minute and think how code might look without Iterables.concat.

for (Item item : Iterables.concat(books, electronics)) {
  // do something useful
}

Multimaps
Multimap is like a Map, but allow multiple values to be stored for every key. The following example is a a variant of a typesafe hetereogeneous container using multimap to realize a product catalogue of items.

public class ProductCatalogue {
  private Multimap<Class, ? extends Item> catalogue = ArrayListMultimap.create();
 
  public void add(Item item) {
    catalogue.put(item.getClass(), item);
  }
 
  public <T extends Item> Collection<Item> list(Class<T> clazz) {
    return catalogue.get(clazz);
  }
}
 
ProductCatalogue catalogue = new ProductCatalogue();
catalogue.add(new Book("1", "Book1"));
catalogue.add(new Movie("2", "Movie1"));
// only get books
System.out.println("Books " + catalogue.list(Book.class));
// only get movies
System.out.println("Movies " + catalogue.list(Movie.class));

BiMap
BiMap implement a one-to-one bidirectional relationship between key and value of the Map. Here is an example using language code to get the language and vice versa.

BiMap<String, String> languageCodes = HashBiMap.create();
languageCodes.put("en", "English");
languageCodes.put("fr", "French");
languageCodes.put("zh", "Chinese");
assert "English" == languageCodes.get("en");
assert "en&amp" == languageCodes.inverse().get("English");

Preconditions
Most classes have restrictions on values given them in constructor and methods. Invalid values should be escalated as soon as possible by doing explicit validity checks before execution. It is a lot better to fail-fast than to fail later with an unexpected exception or worse, silently compute the wrong result.

public Item(String id, String name) {
  this.id = Preconditions.checkNotNull(id, "id must not be null");
  this.name = Preconditions.checkNotNull(name, "name must not be null");
  Preconditions.checkArgument(name.length() < 6, "name must be longer than 6 chars");
}

Constraints
Constraints are similar to preconditions in a way that they can restrict what values are added to a collection. This makes collections much easier to use and code a lot cleaner, since constraints are separated from business code.

public class Voyage {
  private Country targetcountry;
  private int capacity;
  private List<Cargo> items = Constraints.constrainedList(new ArrayList<Cargo>(), new Constraint<Cargo>() {
    @Override
    public Cargo checkElement(Cargo cargo) {
      Preconditions.checkNotNull(cargo);
      Preconditions.checkArgument(targetcountry.allows(cargo));
      Preconditions.checkArgument(cargo.getUnits() > 0);
      return cargo;
    }
  });
 
  public void load(List<Cargo> cargos) {
    items.addAll(cargos);
  }
}

Predicates and Functions
Predicates evaluate if something is true or false but can also be combined into more complex evaluations using “and”, “or”, “not” and “in”.

What normally would require a for-loop and bunch of if statements can now be reduced to a one-liner. How sweet is that?

Predicate<Item> heavyItemPolicy = new Predicate<Item>() {
  @Override
  public boolean apply(Item item) {
    if(item.getWeight() > 1000){
      return true;
    }
    return false;
  }
};
Collection<Item> heavyItems = Collections2.filter(order, heavyItemPolicy);

You can also use Maps.filterKeys or Iterables.filter in a similar way. But keep in mind that removal from modification is bidirectional. e.g. removal from the origin affect result and vice versa.

Functions on the other hand, is a way of transforming one object to another. For example, convert concurrency on a order of items.

Function currencyConverter = new Function<Double, Item>() {
 
  @Override
  public Double apply(Item item) {
   return item.getPrice() * ANOTHER_CURRENCY;
  }
}
Collection<Double> prices = Collections2.transform(order, currencyConverter);

You can also use Maps.transformValues or Iterables.transform in a similar way.

A Query API

I have been think about how to create simple but powerful Fake Objects for some time now. But I dont want fakes themselves to turn into a maintenance burden, so they must be easy to implement. My intuition tells me i need a general purpose state management framework for this to work. And so using predicates, I created a small fluent query interface interacting with an in memory storage.

InMemoryStorage storage = new InMemoryStorage();
// add a few Item.class objects to storage
Criteria middleprice = field("price").is(largerThan(100)).and(lessThan(200));
Criteria expired = field("expires").is(after(currentDate));
Collection<Item> result = storage.select(middleprice.and(not(expired))).from(Item.class);

I feel quite satisfied with the result actually – short, compact, understandable and typesafe.

Im not going to go into the details here, but please do inspect the implementation of Criteria and InMemoryStorage, as well as the tests.

I hope these examples will trigger you to explore Guava further and use it to make your code more readable, robust and maintainable.

And lastly, I really do hope many of these facilities reach standard Java some day soon.

The NetBeans team just got themselves in a really tricky situation when switching to Java 7.

It turns out the that the Class.getMethods() in Java 7 does not return the methods in a deterministic, but random, order. This causes JUnit tests to be shuffled around for every execution. This may be fine, but the NetBeans team have unknowingly introduced tests that are dependent on each other, and the order in which they are executed, causing their test suite of 8000 tests to fall like a domino.

This is an excellent example of the type of maintenance nightmare that I have warned about earlier. I certainly don’t blame them, they hardly could have anticipated this to happen. I have personal suffering from this myself and know how hard it can be to write independent tests. But this incident convince me even more about the fact that tests should be focused, robust and self-contained. I actually already wrote this down earlier in Open Config Developer Guidelines.

Jaroslav Tulach (who btw also wrote this excellent book) explain the problem in more detail here and you can follow to discussion between him and the JUnit crew on their mailing list.

Offshore outsourcing is a complex coorporate undertaking that brings a lot of problems along with it and somehow I have a hard time understanding the logic behind it.

Sometimes it seems as if hourly rate is all that matters, thinking that “code-like-hell” mentalities lowers costs, including the belief that more hands are more productive than few. Tell me, are we still not pass this old-fashioned mindset?

This kind of reasoning is an cheapening insult to the craft of software developers.

I realize that developers carry some responsibility for delivering results. I have had my share of daily WTF projects with people who not truly love or take their profession seriously. But software projects does not usually fail because of this, they fail mostly because of overly optimistic schedules and expectations.

Software development is largely an heuristic process of iteratively massaging and discussing goals and constraints of a problem domain, that gradually spirals towards deeper understanding and ultimately solved from a certain perspective. These continuously gained insights should be shared to all people involved in the project. The most efficient way to exchange such information is face-to-face communication.

So how can Open Source be successful being such a geographically distributed way of doing development? For one, it does not encourage “code-like-hell” mentalities and developers on these projects are usually driven by higher ideals and motives that magnetize quality. This is not my point.

Please do excuse a caring software developer’s honest and frustrated reflection, merely scratching the surface, there a lot of aspects to the equation and I do not claim to have a solution. But outsourcing core business without accompanying a comprehensive and technical argumentation for why this decision is taken seeds feelings of disappointment.

Take a step back and ask yourself, is it possible for all of us to focus on maximizing value instead of being obsessed with minimizing cost, or have top management already given up on quality?

Henry Thoreau said, “we become tools of our tools”, which basically means that the more complex our tools are, the more complex our lives become.

Less is more, remember?

I have noticed that people sometimes avoid doing thorough testing. This may sound as bogus to some, but hear me out… I do understand why this happen.

Tests can create a feeling of being trapped, a burden that gets heavier to carry with every new test introduced. Building a stable, non-intrusive and quality ensuring suite of tests is a difficult task… but *why* does these problems surface?

Most people can agree that testing, performed in one way or another, for any type of product, is a great way for revealing quality and potentially later increase it. But the testing process can get problematic when you specify a static suite of tests and then continuously re-execute it through the lifecycle of the product, i.e. regression testing.

Stop and think for a second… what value does interfaces and abstractions provide to clients? They provide a way to enjoy a *particularly* valuable service without having to worry about the complex internal details of *how* it is provided to us.

Interfaces are not unique to software development, they are everywhere in our everyday community. Consider leaving your Toyota in for repair. Employer Mike is excellent fixing Toyota cars using custom-made tools. But do you, as a client, really care *how* and by *whom* the car was repaired, if you can observe that malfunctioning parts have been replaced and the car works better than before?

Mike will do his job better without being bothered by people who cannot keep up with his bleeding-edge techniques. Mike may quit or learn even better methods, throwing some of his old custom-made tools away. It is better to assure that the company service, as a whole, can deliver high quality for a wide array of diverse problems and clients.

This is a simplification, but you get the point.

I have observed that testing efforts (far too often) tries to verify unspecified implementation details, without really thinking of the consequences. Having too many poorly specified tests can easily snowball into a chaotic maintenance nightmare, making future product development difficult and unproductive. When these symptoms come creeping developers stop making continuous improvements, to avoid problematic refactoring and infectious conflicts with QA, hurting product quality more than helping it.

Internal domain experts can help you understand fundamental business strategies and what *really* matters for the company to be successful. Think long and hard when specifying significant interfaces that are visible for *both* internal and external business processes. Distill abstractions that capture high-level goals and concepts, not complex implementation details. Focus your design and testing efforts on these interfaces and you will earn stability.

Remember that business is fiercely competitive and vibrant, changing rapidly to outrun competitors. Allow flexibility for business to grow and evolve when defining interfaces and tests in your architecture. Enable interfaces to be combined into unique service compositions to support unforeseen business capabilities, maximizing value with minimal efforts.

The future will forever remain untold so do not make too many assumptions on what tomorrow brings.

There is no such thing as a crystal ball.

I have spent some time writing a foundation for Open Config Developer Guidelines. At the moment this is a very early draft summary of my own experience, gained through years as a software developer. A lessons learned if you will. I do hope for more people helping me on this mission

Why are developer guidelines necessary? At first it may seem as a smart-ass/dictator thing to do, trying to write people on their nose. This is far from the truth.

First, coding is a team effort. Developer guidelines are the pride and identity of the team, explaining their philosophy for collaboration and getting things done.

Second, the power lies in hands of the team, free for developers to adopt their own styles and change these guidelines together, as a team. There is no such thing as “dictatorship” in this regard, tyranny and arrogance pretending to be a democracy will turn people off.

The purpose of developer guidelines is to encourage the team to maintain a certain level of consistency and quality throughout all aspects of the project. Having different ways and opinions of achieving similar things cause confusion, inconsistency, and in the end, segregation within the team. This must be avoided, the team cannot afford loosing focus and good contributors in frustration and agony.

Do notice the emphasis on “guidelines” here, there will always be exceptions to every rule. Developers are expected to be thoughtful enough to care and take appropriate action (within reasonable boundaries) when these do not apply, maybe providing notice to the team of anomaly or ambiguity.

Individuals should not be afraid their opinions. But sometimes this cause strong philosophical disagreement to occur within the team. When this happens, the team accomplish consensus through voting. Individuals are expected to show good judgement resolving such conflicts, providing relevant justificating arguments from their point of view. If the conflict is resolved favoring a change of direction, individuals +1 voting for the proposal are expected to help making the change a reality.

The satisfaction of creativity is an energising motivator not to be underestimated. The team should feel proud of their creations, not frustrated by overly-constrainting rules.

This is work in progress and I sincerely invite anyone to participate. Comments and feedback on latest Open Config Developer Guidelines are more than welcome!

Ever had performance problems? Yeah me too. If my manager screams “faaaaster” one more time, i will have hearing impairment for the rest of my life. BTW, did i sense a german pronunciation in all that noise?

Can you believe that there are still people doing ignorant trash talk about the garbage collector (get it?) and performance of the JVM..

… i will go back writing C again so i dont have to worry about performance…

*sigh*

The JVM is continuously improving its collector algorithms and highly sophisticated optimizations are incorporated into the compiler with every release (and have been doing so for the last 10 years). Do *you* really expect to have the experience, ability and time to write better and more optimized C code than some of the smartest people on this earth?

Pleeeeease..

If you are like me and 99.99 percent rest of us, you be wise to forget about C. Just get over it. (salute to all hardcore C programmers, do not feel provoked)

As much as us developers love abstractions, we cannot deny the fact that they are inherently leaky. Hardware *does* matter. The trend of processor count and memory growth make shared memory thread concurrency a lot harder. Locking, context switching and thread scheduling can make your throughput equal to syrup, thinking that pouring more threads into your shiny new super-beefy-machine will somehow magically will give you more performance. It probably will to some degree, but that’s not my point.

So what to do? I do not claim to be a performance expert, i am not, but i have some practical advice that at least helped me squash some nasty performance bugs in the past.

1. Write clean and “dumb” code. Consider making your classes immutable, they are thread-safe hence no need for synchronization and can be cached with confidence that object values do not change after being created. Immutability also leads to code that is easier to understand. Do not try to outsmart the JVM with premature optimization tricks.
   
   
   Donald Knuth said: “Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.“




2. Spend some time on understanding the how different garbage collectors works. The information is a bit scattered, but its out there. Find the resource sharing sweet-spot between garbage collection and your application. Generally speaking larger heaps means garbage collector needs to work harder (stealing more CPU cycles) and pauses will be longer, but less frequent. In my experience you cannot avoid stop-the-world pauses from happening, even using CMS, because eventually your heap will be fragmented as swiss cheese, and boom, memory fragmentation failure. The good news is that JDK7 will probably include a new low-pause collector, called G1, which can potentially fully avoid stop-the-world pauses. UPDATE: See Grabage-First Collector(G1) in Java 7.

.




3. Always use java.util.concurrency by default when programming. Read the Java Memory Model and Thread Specification. It will help you understand why your code may not performing as it should. There are very good books on the subject of concurrency aswell:
   • Java Concurrency in Practice
   • The Art of Multiprocessor Programming
   • Concurrent Programming in Java: Design Principles and Pattern (2nd Edition)

   
   

4. Chances are that you are dealing with legacy code (you cannot influence) that have coarse grained synchronization, causing high thread contention. Using CPU affinity with multiple JVM processes on the same machine can help reduce contention for hot locks.




5. If you think you found JVM performance problems by doing benchmarks, first, make sure you know that you *know* your measurements are accurate. If you try measure something, dont measure other stuff. Ignoring this advice may mislead you from where the real problems lurk. So make sure to properly isolate system parts before you start measuring.

For example, if you suspect thread contention, have a look at ThreadInfo or try jstat and look for sun.rt._sync_ContendedLockAttempts.
[java]jstat -J-Djstat.showUnsupported=true -snap PID | grep _sync_[/java]

There are so much to say on this subject, but i dont have time to write more right now. Happy coding!

Documentation is something that most developers fear/hate and some of us usually avoid it til the end of a project. And im not talking about, the more mandatory, javadoc. *Ahem*

However, open-config need a presentation sketch for readers (and potential contributors) to learn about the project, allowing them to form an immediate first impression.

I dont want (and cant) document everything before even getting started. But the plan is to have a clear and concise mission statement, brief list of planned features, the most fundamental development guidelines and principles, high level design and something about project governance. I will do this before start coding, because I do not want to worry about it (too much) when im “in the zone” .

I recently found out about docbook, which btw is highly praised by the open source community, im actually a bit surprised i did not stumble on it in earlier. Docbook allow writing content decoupled from the presentation-view, using XML files. These XML files can then be transformed into navigable HTML pages, PDF, XHTML, man pages etc. Sweet! The syntax is a bit complex but since it is based on regular XML i can probably use an editor, with autocompletion and nice GUI, to help me out.

Docbook can also be easily integrated into maven (well, i had to spend a few hours on it) using docbkx. This is important because documentation must be tracked by strict version control, a wiki is far too weak in this regard. Having documentation in the same GitHub repository as code makes it easy to update since commits can include documentation changes and thus remain up-to-date with latest development. Documentation can also be branched along with code releases.

When changes are made, simply upload the latest maven-generated HTML and PDF to the project website.

The most important project artifacts can now stay in (almost) perfect sync, no matter what medium readers wish to view it on. No more excuses for doing late documentation… yikes!