ADLB Performance Counters

ADLB’s performance counters let you get useful summary statistics about what the servers were up to without imposing much overhead. They’re handy to turn on for large-scale runs where you want to understand performance but don’t want to be slowed down by extensive logging.

Each ADLB server prints it’s own set of counters are the end of execution.

ADLB Operation Counters

One set of performance counters records the incoming requests handled by an ADLB server. This can help understand where an ADLB server spends it’s time, as each request takes very approximately the same amount of time to process. The ADLB servers can sustain quite high operation throughput - on a circa-2012 Intel desktop, 200k-300k operations per second is typical.

Task operations:

GET: Blocking task get requests, including data notifications

IGET: Non-blocking task polling

PUT: Task adding to queue

Data operations:

Creation:

CREATE_HEADER: create a single variable

MULTICREATE: create multiple variables at once

Storing:

STORE_HEADER: store data values (including to container subscripts)

INSERT_ATOMIC: try to insert a nested container

Retrieval/info:

RETRIEVE: retrieve data (whole data structure or subscript)

ENUMERATE: enumerate container contents

CONTAINER_SIZE: size of container

EXISTS: check if id/subscript exists

TYPEOF, CONTAINER_TYPEOF: get data types

Notifications:

SUBSCRIBE: notification when id or id/subscript closed

CONTAINER_REFERENCE: notifications for containers

Misc:

REFCOUNT_INCR: any reference count modifications

LOCK, UNLOCK: locking a data element

UNIQUE: get the next unique data id

Admin traffic

SYNC_REQUEST: server-to-server sync when server needs to interact with another server. Syncs may be followed by another operation, or they may be a work stealing probe.

CHECK_IDLE: checking for shutdown condition

SHUTDOWN_SERVER, SHUTDOWN_WORKER: sent by master server to do clean shutdown

FAIL: only upon errors

Task Counters

Another set of performance counters records statistics about the tasks going through the system. They are categorized in several dimensions:

There are three non-overlapping categories:

single: regular untargeted single-worker tasks

parallel: parallel tasks

targeted: targeted tasks (usually data notifications)

Each of these is divided by the ADLB work type tag:

worktype_0: worker tasks

worktype_1: control tasks and notifications

We collect several statistics:

net: tasks matched by this server. Summing across all servers gives the total number of tasks that went through ADLB

total: total tasks handled by this server. If you sum across servers, stolen tasks will be double-counted

bypass: tasks that were matched immediately to receiver and not enqueued

enqueued: tasks that had no immediate receiver: added to queue on server

stolen: number of tasks stolen from this server

Magical Function Annotations

These function annotations give the compiler information about functions so that it can do amazing things like optimize code. Without annotations it is forced to assume that the function can do whatever it feels like, such as returning non-deterministic values, doing I/O, or other nasty things that make it difficult to optimize code. If you’re not sure whether you should add an annotation to your function, then the safe thing is not to do it.

@pure: this tells the compiler that the function is deterministic and side-effect free.

@dispatch={WORKER|CONTROL|LOCAL}: the function should be executed on a worker process/control process/the local process. This has no effect if we’re calling a user-provided wrapper function.

@assertion: the function should be treated as an assertion. It will be lexically removed when the disable-asserts option is enabled

@builtin_op=OP_ENUM_VALUE: this is a special hook that tells the compiler that the function corresponds to a special operation that it knows about. It can then do wondrous things like evaluate the function at compile time. Unless you are working on compiler internals, you have no need to use this.

@commutative: arguments can be reordered without affecting result

@minmax: function evaluates minimum or maximum