TOP CPU consumers on Billion Scale

[filipecosta90]

collected via:

py-spy top --pid 99870 -s -r 999

Collecting samples from 'python3 run.py --host 192.168.2.6 --engines redis-intel-float16-hnsw-m-16-ef-32 --datasets laion-img-emb-768d-1Billion-cosine --skip-search --upload-start-idx 630000000 --upload-end-idx 640000000' and subprocesses
Total Samples 7500
GIL: 95.00%, Active: 191.40%, Threads: 16, Processes 11

  %Own   %Total  OwnTime  TotalTime  Function (filename)                                                                                                                                                           
 42.80%  42.80%    3.21s     3.21s   _send (multiprocessing/connection.py)
 27.40%  27.40%    2.11s     2.11s   _read_from_socket (redis/_parsers/socket.py)
 13.30%  13.30%    1.13s     1.13s   _recv (multiprocessing/connection.py)
 12.10%  20.00%    1.07s     1.73s   get (multiprocessing/queues.py)
 14.00%  71.60%    1.05s     5.60s   upload_batch (clients/redis/upload.py)
  8.70%  12.20%   0.659s    0.890s   dumps (multiprocessing/reduction.py)
  7.70% 110.30%   0.607s     8.72s   worker (multiprocessing/pool.py)
  7.20%   7.20%   0.524s    0.524s   poll (multiprocessing/popen_fork.py)
  5.80%   5.80%   0.405s    0.405s   __enter__ (multiprocessing/synchronize.py)
  3.50%   5.10%   0.348s    0.406s   read_data (dataset_reader/ann_h5_multi_reader.py)
  3.20%   3.30%   0.242s    0.248s   send_packed_command (redis/connection.py)
  3.20%   3.20%   0.218s    0.218s   __init__ (multiprocessing/reduction.py)
  3.20%   3.60%   0.217s    0.252s   pack (redis/connection.py)
  2.10%  58.90%   0.179s     4.50s   _handle_tasks (multiprocessing/pool.py)
  1.70%   8.00%   0.178s    0.665s   _determine_nodes (redis/cluster.py)
  2.20%  56.90%   0.172s     4.48s   execute_command (redis/cluster.py)
  1.50%  45.70%   0.136s     3.58s   _execute_command (redis/cluster.py)
  2.10%   2.10%   0.131s    0.140s   get_keys (redis/_parsers/commands.py)
  1.20%  44.00%   0.122s     3.33s   _send_bytes (multiprocessing/connection.py)
  1.60%   2.00%   0.114s    0.146s   register (selectors.py)
  0.90%   7.90%   0.089s    0.541s   get_connection (redis/connection.py)
  1.00%   3.60%   0.089s    0.268s   send_bytes (multiprocessing/connection.py)
  1.30%   9.90%   0.082s    0.705s   put (multiprocessing/queues.py)
  0.40%   0.40%   0.079s    0.079s   get_node_from_slot (redis/cluster.py)
  1.10%  72.70%   0.077s     5.68s   _upload_batch (base_client/upload.py)
  0.70%  57.60%   0.076s     4.56s   hset (redis/commands/core.py)
  1.00%   1.00%   0.070s    0.070s   _checkpid (redis/connection.py)
  0.80%   0.80%   0.069s    0.069s   encode (redis/_parsers/encoders.py)
  0.90%   0.90%   0.069s    0.069s   __contains__ (redis/client.py)
  0.70%  27.70%   0.068s     2.19s   parse_response (redis/client.py)
  0.90%   3.30%   0.068s    0.232s   _get_command_keys (redis/cluster.py)
  1.00%   5.90%   0.065s    0.407s   determine_slot (redis/cluster.py)