Understanding Pickaxe Configurations

Listed below is a description and guide related to each Pickaxe configuration.

Important Note:

The default settings are optimized for most use cases.
Uninformed configuration changes can have adverse effects on your fleet.
Do not change your Pickaxe configurations without input from the Foreman team.

To view the Pickaxe configurations in Foreman:

Navigate to the Pickaxe page. https://dashboard.foreman.mn/dashboard/pickaxe/
Click on the gear icon located in the top right corner of your Pickaxe instance.
From the dropdown menu, click "Configure" to access the Pickaxe configuration modal.

The Pickaxe Configuration Modal includes four tabs:

1. Timeouts
2. Threads
3. ENV
4. PPM

The Timeout Configuration tab is tied to how long Pickaxe should wait to read, write, collect and push data.

Socket Read Timeout

Configures the wait time when reading data from a miner.
Increase this value if you experience high network latency or use a VPN.
Default: 1 second.

It is advised to leave this at one second. Increasing it will in turn increase the stats iteration time, scan time, and resync time. Most miners respond in less than one second.

Socket Write Timeout

Configures the wait time when running commands against a miner.
Default: 20 seconds.

This value should typically never be higher than 20 seconds.

Miner Collect Delay

Configures the delay between miner metrics collections in the LAN.
Default: 30 seconds.

This is dependent on how long it takes to collect the stats. If the stats collection takes longer than the configured amount in the miner collection delay, then no delay occurs. Stats will run constantly with no delay.

Miner Push Delay

Configures the delay before publishing metrics to the Foreman API.
Never set it to lower than 1 minute.
Default: 1 minute.

This configuration is tied to how long Pickaxe will wait to send the data to Foreman after the data has been collected. If you have it set to the default setting this also means there is a one minute delay when seeing stats appear in the Foreman dashboard.

Time series data within the dashboard can be affected when this is set above 2 minutes. Generally there’s no reason to modify this value from the default of 1 minute.

Infrastructure Collect Delay

Configures the delay between miner infrastructure-related metric collections.
Default: 90 seconds.

This default configuration should not be adjusted.

Infrastructure Push Delay

Configures the delay before publishing infrastructure-related metrics to the Foreman API.
Default: 2 minutes.

This default configuration should not be adjusted.

Command Batch Size

Configures the number of command completions pushed to the Foreman API at once.
Default: 500

This is the max amount of command updates that are sent at a time from Pickaxe with the overall purpose to reduce the number of network calls that are made to the Foreman dashboard.

This value should be increased if you'll be frequently issuing thousands of commands at once.

Miner Batch Size

Configures the number of metrics batched before sending to the Foreman API.
Default: 400.

This is used to regulate how large the network request is when sending metrics.

Infrastructure Batch Size

Configures the number of infrastructure-related command completions pushed to the Foreman API at once.
Default: 200.

This default configuration should not be adjusted.

The Threads configuration tab provides controls to facilitate efficient and concurrent processing of tasks.

The Pickaxe thread logic is based on the number of processors the Pickaxe has. For reference, most users operate with 8 or fewer processors.

Setting all the threads to 0 by default allows Pickaxe to operate with optimal thread values.

Stats Threads

Configures the number of threads used to query miners.
Adjust based on network bandwidth to optimize performance.
Default Stats is determined by the number of processors x 32 or 300, whichever is greater

This is used as a way to rate limit Pickaxe with the network. More threads doesn't mean better performance, it just means more threads. Depending on your network bandwidth, less might be more. In some cases, a decrease in stats threads might make the process more efficient, while maintaining iteration time and decreasing strain on the network.

A decrease in stat threads should be considered when Pickaxe is taking up too much network bandwidth. Generally lower stats threads allows for more command threads, which can decrease the time it takes to do bulk commands.

Ideally on any given network, iteration time should be under 60 seconds. If your iteration time is over 90 seconds an increase in Stats threads is recommended.

Stats Push Threads

Configures the number of threads for pushing metrics to the Foreman dashboard.

The total number of requests made to report stats is determined by dividing the total number of miners by the “Miner Batch Size” property. The “Stats Push Threads” determines how many of those requests are made at the same time.

We recommend you do not change this configuration.

Infrastructure Threads

Configures the number of threads to query infrastructure-related items.

This does not need to be adjusted under any circumstances.

Infrastructure Push Threads

Configures the number of threads for pushing infrastructure metrics to the Foreman dashboard.

This does not need to be adjusted under any circumstances.

Command Threads

Configures the number of threads for processing commands.
Adjust based on network bandwidth and the number of miners.
Default Command is determined by the number of processors x 8

This is related to how many commands Pickaxe can process in parallel. Depending on your network bandwidth and the total number of miners running on your instance of Pickaxe, adding more threads may or may not translate to an increase in efficiency.

An increase in command threads could potentially cause more strain on your network. If command processing needs to be faster and increasing this property does not help, consider lowering the stats threads as long as your iteration time is under 60 seconds.

Scan Threads

Configures the number of threads for scans to find new miners.
Default Scan Threads is determined by the number of processors x 8 or 100, whichever is greater

Ideally the lower the better for this. An increase in scan threads can cause network strain.

Resync Threads

Configures the number of threads for resync scans to find existing miners.
Default Resync threads is determined by the number of processors x 8 or 100, whichever is greater

Resyncs should take less than two hours and increasing this number can reduce that total time. However, increasing the number of resyncs can add strain to the network thus slowing down stats and command processing. If your read timeout is over 1 second, consider lowering the read timeout since some of the IPs scanned in a resync will not have a miner, which could use the entirety of the read timeout for each call Pickaxe attempts.