Installation¶

If you are in a Jupyter Notebook, you can install comet_ml directly in the notebook by using this magic:

%pip install comet_ml --upgrade --quiet

If you have a complex Python environment, you can use this format to install comet_ml at the command line:

bash /path/to/python -m pip install comet_ml

Other useful pip variations at the command line:

python3 -m pip install comet_ml --user : using Python 3, installs comet_ml to ~/.local
pip install comet_ml --upgrade : using the default Python, upgrades comet_ml to latest version
pip3 install comet_ml --upgrade --upgrade-strategy eager : using Python 3, upgrades comet_ml and all of its dependencies

The flags --user, --upgrade, and --upgrade-strategy can be used in many combinations.

Info

If you're installing the SDK on shared servers or as part of a CI/CD system it's encouraged to use the --upgrade flag for pip as our SDK is constantly evolving.

If you are using conda or would like to log CPU usage, please see our additional Installation Instructions.

Python Configuration¶

In order to interact with Comet.ml you will need to let it know what your Comet API key is. Recall that you can get (and reset) your Comet API key via this URL:

www.comet.ml/api/my/settings

If using an onprem version of Comet, please adjust the URL for your installation location.

Interactive Setup¶

If you are in an interactive environment (such as a Jupyter Notebook) connected to the internet, the easiest way to get started is the following lines in a cell near the top of the notebook:

python import comet_ml comet_ml.init()

The first time you use this it will ask you for a valid Comet API key, and save it in your ~/.comet.config file (see below). Subsequent uses will not ask for your Comet API key.

In addtion, you can temporarily override Comet configuration options, like this:

python import comet_ml comet_ml.init(project_name="nlp-1267")

In addition setting and saving the API key, this variation will temporarily set the COMET_PROJECT_NAME to "nlp-1267" so that all of the Experiments created in this session will be logged in that project. See Experiment Configuration Parameters below for additional configuration options you could set in this manner.

You can also run comet init --api-key at the command-line to set and save the Comet API key. See Comet Command-Line Utilities for additional information.

Non-interactive Setup¶

There are a number of methods for you to set your API key prior to running a script. They are in order of highest precedence:

An argument passed to Experiment("YOUR-API-KEY")
An environment variable named COMET_API_KEY
A setting in the .comet.config file in your current folder
A setting in the .comet.config file in your HOME directory

That is, an explicit API key passed to Experiment() has top priority, and would override any other method. To get started immediately, you can use the following:

python import comet_ml experiment = comet_ml.Experiment("YOUR-API-KEY") Alternatively, an environment variable named COMET_API_KEY has the second-highest priority, and would override any configuration file setting. In a Unix-like environment, you could set it like this:

bash export COMET_API_KEY=”YOUR-API-KEY”

However, in general, we recommend using a file named .comet.config (note the initial dot in the filename) in your home directory. That way, it will always be found, and you'll never have to set it again. The file format of the .comet.config file is as follows:

ini [comet] api_key=YOUR-API-KEY

After you create the file, you can then create an Experiment() without having to put your private Comet API key in your source code.

You can get (or reset) your API key under your personal settings at www.comet.ml/api/my/settings.

NOTE: the COMET_REST_API_KEY has been deprecated. You can now simply use your COMET_API_KEY. For more information, please see REST API, Getting Started.

Debugging with Comet¶

When you run a Python script creating Experiments with comet_ml, your script will run as it normally would. That is, even if you, say, attempt to log something invalid, your script will run without interruption to completion. Let's take a specific look at this case, and explore how to debug it.

Consider a script that attempts to log an invalid image:

```python

filename: script.py¶

from comet_ml import Experiment

e = Experiment()

image = [] e.log_image(image)

further processing...¶

print("Done!") ```

This script will run without stopping or crashing even though the image is not a proper image. Here is the output of running the script:

$ python script.py ... COMET ERROR: Could not convert image_data into an image; ignored Done! ...

For many cases, the ERROR message might be enough of a hint to know what to fix. But what if it isn't obvious why the image didn't log? You have two options:

Create a Comet Debug Log file. This is especially useful to send to Comet for detailed analysis. See Troubleshooting below on how to create a detailed log file.
Turn on tracebacks while you run the script.

The second option will run your script exactly as before, but will also show any tracebacks from comet_ml operations that failed. It won't stop processing, but it will show the traceback of any errors. Because it could be confusing to see a traceback without crashing, we don't show them by default. But it is easy to enable this setting:

You can set the COMET_LOGGING_CONSOLE on the command line, like:

COMET_LOGGING_CONSOLE=info python script.py

or:

export COMET_LOGGING_CONSOLE=info python script.py

When you run the script this way, you will see output similar to:

$ COMET_LOGGING_CONSOLE=info python script.py ... COMET ERROR: Could not convert image_data into an image; ignored Traceback (most recent call last): File "/home/dblank/comet/python/comet-client-lib/comet_ml/file_uploader.py", line 659, in process_upload_to_be_converted self.image_channels, File "/home/dblank/comet/python/comet-client-lib/comet_ml/utils.py", line 284, in image_data_to_file_like_object image_channels, File "/home/dblank/comet/python/comet-client-lib/comet_ml/utils.py", line 322, in array_to_image_fp minmax = [array.min(), array.max()] File "/home/dblank/.local/lib/python3.6/site-packages/numpy/core/_methods.py", line 34, in _amin return umr_minimum(a, axis, None, out, keepdims, initial, where) ValueError: zero-size array to reduction operation minimum which has no identity Done! ...

As you can see, it shows the stacktrace that caused the problem, but also still continues with processing.

You can also set the COMET_LOGGING_CONSOLE variable in your Comet Configuration file as described in the following section.

Comet Configuration Variables¶

Here is a list of the main configuration variables (followed by details on specific sections):

Configuration item	Comet Config Section/Name	Environment variable	Description
API key	[comet] api_key	COMET_API_KEY	The API key used for creating Experiments and using the Python API
Current Workspace	[comet] workspace	COMET_WORKSPACE	Use this workspace when creating new experiments
Current Project	[comet] project_name	COMET_PROJECT_NAME	Use this project when creating new experiments
Offline Directory	[comet] offline_directory	COMET_OFFLINE_DIRECTORY	Set the offline directory for `OfflineExperiment()`
Git Directory	[comet] git_directory	COMET_GIT_DIRECTORY	Set the directory that contains your git repo, if you are not under it
Display Summary	[comet] display_summary	COMET_DISPLAY_SUMMARY	If False, do not display upload summary for experiments
Hide API key in logs	[comet_logging] hide_api_key	COMET_LOGGING_HIDE_API_KEY	If True, redact the Comet API key in logs
Logging file	[comet_logging] file	COMET_LOGGING_FILE	Use the given file for storing log messages. This can also contain patterns such as "comet-{project}.log" (See below)
Logging file level	[comet_logging] file_level	COMET_LOGGING_FILE_LEVEL	By default, the log outputs will contains INFO and higher level (WARNING and ERROR) log messages. This configuration item can be used to change the level of the logged messages.
Logging file overwrite	[comet_logging] file_overwrite	COMET_LOGGING_FILE_OVERWRITE	Overwrites the log file on each run, if True
Console logging	[comet_logging] console	COMET_LOGGING_CONSOLE	Set the logging level for console messages (e.g., INFO, DEBUG)
Logging auto values	[comet_logging] metrics_ignore	COMET_LOGGING_METRICS_IGNORE	List of framework:metric names to not auto-log
Logging auto values	[comet_logging] parameters_ignore	COMET_LOGGING_PARAMETERS_IGNORE	List of framework:parameter names to not auto-log
Logging auto values	[comet_logging] others_ignore	COMET_LOGGING_OTHERS_IGNORE	List of framework:other names to not auto-log
Experiment end timeout	[comet_timeout] cleaning	COMET_TIMEOUT_CLEANING	How many times (in seconds) does the Python SDK wait until all experiment data has been uploaded. Default is 3600 seconds (i.e. 1 hour).

Experiment Configuration Parameters¶

In addition to the above, you can also control exactly what each experiment logs with the following configuration variables in the section [comet_auto_log]:

Experiment Loggable Item	Comet Config [comet_auto_log] Section Name	Experiment parameter	Environment variable
code, Jupyter Notebook	code	log_code	COMET_AUTO_LOG_CODE
git metadata (log_code must be True)	git_metadata	log_git_metadata	COMET_AUTO_LOG_GIT_METADATA
git patch (log_code must be True)	git_patch	log_git_patch	COMET_AUTO_LOG_GIT_PATCH
model graph representation	graph	log_graph	COMET_AUTO_LOG_GRAPH
model parameters	parameters	auto_param_logging	COMET_AUTO_LOG_PARAMETERS
model weights and biases	weights	auto_weight_logging	COMET_AUTO_LOG_WEIGHTS
training metrics	metrics	auto_metric_logging	COMET_AUTO_LOG_METRICS
command-line arguments	cli_arguments	parse_args	COMET_AUTO_LOG_CLI_ARGUMENTS
console and Jupyter Notebook standard output and error	output_logger	auto_output_logging	COMET_AUTO_LOG_OUTPUT_LOGGER
CO2 emissions	co2	auto_log_co2	COMET_AUTO_LOG_CO2
environment details	env_details	log_env_details	COMET_AUTO_LOG_ENV_DETAILS
environment GPU	env_gpu	log_env_gpu	COMET_AUTO_LOG_ENV_GPU
environment hostname	env_host	log_env_host	COMET_AUTO_LOG_ENV_HOST
environment CPU	env_cpu	log_env_cpu	COMET_AUTO_LOG_ENV_CPU
everything; if True, everything above is disabled	disable	disable	COMET_AUTO_LOG_DISABLE

Remember: any configuration setting defined as an environment variable will override the value set in the .comet.config file.

The auto_log_co2=True setting automatically tracks the CO2 emission of the experiment if the codecarbon package is installed in the environment. For more details please see Introducing CodeCarbon.

Comet File Logging¶

As mentioned, you can use a number of patterns in the COMET_LOGGING_FILE name ([comet_logging] file value). These are keywords surrounded by curly braces as follows:

COMET_LOGGING_FILE Pattern	Example	Expanded Form	Meaning
{user}	comet-{user}.log	comet-jones.log	Uses the OS username
{project}	comet-{project}.log	comet-general.log	Uses the COMET_PROJECT_NAME if set, else "general"
{pid}	{project}-{pid}.log	general-2635.log	Uses the OS Process ID
{datetime}	comet-{datetime}.log	comet-20190814-151135.log	Uses a Year, Month, Day, Hour, Minute, Seconds time format

Other Configuration Logging¶

Configuration item	Comet Config Section/Name	Environment variable	Description
CPU/GPU system prefix	[comet] distributed_node_identifier	COMET_DISTRIBUTED_NODE_IDENTIFIER	A prefix to be used on all GPU/CPU logged system metrics

There is also an additional flag that can only be set in the environment: COMET_DISABLE_AUTO_LOGGING (see below for addition environment-only settings). Use this to disable Comet’s automatic framework tracking functionality. For a complete list of items logged per Python framework, see Automatic Logging. Comet.ml will not interfere with your run, unless you attempt to use one of the auto logging Python libraries, but you have imported comet_ml after the library. For example:

```python import tensorflow import comet_ml

experiment = comet_ml.Experiment() ```

When you create an Experiment then you will get an ImportError. To fix, simply move the import comet_ml above the other import statement.

In some rare cases, something will fail (such as the connection between your computer and comet.ml). In that case, your code will keep on running even though the run will not be logged with Comet.ml. If you wish for your code to stop if it is not being logged, we suggest the following pattern:

```python from comet_ml import Experiment

exp = Experiment() if exp.alive is False: raise Exception("Something went wrong") ```

That is, after an Experiment has been created, you may check the alive property. If alive is False then you know something has failed, and you could stop your run. Of course, there is also the OfflineExperiment to use on computers with flaky (or non-existent) internet connections.

Comet Environment-Only Variables¶

In addition to the configuration variables, there are a few special-purpose Comet environment-only variables. These only operate in the environment, and cannot be set in the configuration:

Environment Variable	Description
COMET_INI	The path to your `.comet.config` file
COMET_OPTIMIZER_ID	Current optimizer ID; for use with the `comet optimize` CLI command
COMET_OPTIMIZER_PROCESS_ID	Current job number; set when using `comet optimize` CLI command
COMET_OPTIMIZER_PROCESS_JOBS	Total number of parallel jobs; set when using `comet optimize` CLI command
COMET_DISABLE_AUTO_LOGGING	Disable all auto-logging (see next section)

Comet Auto Logging¶

Comet can automatically log certain values when using one of the following machine learning frameworks:

Framework	What is automatically logged
tensorflow	steps and metrics; see examples
tensorflow model analysis	time series, plots, and slicing metrics
tensorboard	summary scalars (as metrics) and summary histograms
sklearn	hyperparameters; see examples
keras	graph description, steps, metrics, hyperparameters, weights and biases, activations, and gradients as histograms, optimizer config, and number of trainable parameters; see examples
mlflow	hyperparameters, assets, models, plus lower-level framework items (e.g., tensorflow's metrics, tensorboard summaries)
fbprophet	hyperparameters, model, and figures
pytorch	graph description, steps, and loss; see examples
pytorch-lightning	loss and accuracy, with refinements in progress; see examples
pytorch + apex	loss; see examples
fastai	all pytorch items, plus epochs, and metrics; see examples
xgboost	metrics, hyperparameters; see examples

See the framework page for more details on its specifics.

Info

Also check out the comet init command-line method of creating best-practices code with Comet.

However, there may be items that are logged that you wish to ignore, or items ignored that you wish to log. You can control exactly what is auto-logged using the appropriate logging config variables. Each of the ignore variables should be set to a list of framework:name items, separated by commas. Here is an example showing the default values:

ini [comet_logging] metrics_ignore = keras:batch_size,keras:batch_batch others_ignore = parameters_ignore = keras:verbose,keras:do_validation,keras:validation_steps

The metrics_ignore line indicates that the metric keras:batch_size will not be auto-logged to Comet. The reason that it is not logged as a metric is because it does not typically change over time, and is usually regarded as a parameter. However, if it did change over time, you can remove it from this list and it then would be logged to Comet.

When you run a script using Comet, it will print out any auto-logged framework:name that is ignored. You can then adjust these configuration variables so that you have complete control over what is logged.

HTTP Proxy¶

You can instruct comet_ml to use an HTTP proxy by setting one of the common HTTPS variables in an appropriate manner for your operating system. Comet will look for environment variables named HTTPS_PROXY or https_proxy, for all operating systems first, and if it cannot find them, it looks for proxy information from Mac OSX System Configuration for Mac OS X and Windows Systems Registry for Windows. If both lowercase and uppercase environment variables exist (and disagree), the lowercase is preferred.

You can use any standard format for setting the proxy:

"hostname.com:9999"
"http://hostname.com:9999"
"http://username:password@hostname.com:1080"
"socks4://hostname.com:1080"
"socks5://hostname.com:1080"

For example, in bash you can:

bash export HTTPS_PROXY="http://username:password@hostname.com:22" python train.py

Info

Note that you must use the HTTPS version (not HTTP). Also, to use the socks4 or socks5 protocol, you must have the "PySocks" Python library installed.

You may also define the NO_PROXY environment variable (all capital letters) to define a list of domains that should not be consulted. For example:

bash export NO_PROXY="hostname.com:22,localhost,cern.ch" python train.py

For more information on using proxies, please see Setting Up Clients To Use a Proxy.

Context Managers¶

In addition to the default method of logging shown in Getting Started, you can also use the Python contextual managers with experiment.train(), with experiment.validate(), or with experiment.test() which will allow metrics and hyperparameters will be prefixed accordingly. For example, "accuracy" will be reported as "train_accuracy" in the with experiment.train() context manager. Here is an example using all three context managers:

```python import comet_ml experiment = comet_ml.Experiment()

model = build_model()

with experiment.train(): train_model(model)

with experiment.test(): test_model(model)

with experiment.validate(): validate_model(model) ```

The Experiment object logs various parameters of your experiment to Comet.ml. For example:

```python from comet_ml import Experiment experiment = Experiment() batch_size = 4 # A hyperparameter used somewhere in the code.

experiment.log_parameter("batch_size", batch_size) ```

By default your experiment will be added to the project for Uncategorized Experiments named general. You can also log your experiment to a specific project using the project_name argument of the Experiment class:

```python from comet_ml import Experiment

if "my project name" does not already exist, it will be created.¶

experiment = Experiment(project_name="my project name") batch_size = 4

experiment.log_parameter("batch_size", batch_size) ``` You can also log a custom list of hyperparameters to your experiment via a dictionary.

```python from comet_ml import Experiment experiment = Experiment(project_name="my project name", auto_param_logging=False) batch_size = 128 num_classes = 10 epochs = 20

params = { "batch_size":batch_size, "epochs":epochs, "num_classes":num_classes }

experiment.log_parameters(params) ```

Reporting from SDK to a workspace project¶

When initializing a team experiment provide the workspace argument. This will assign the experiment to the correct workspace.

```python from comet_ml import Experiment experiment = Experiment(workspace="my-workspace-name", project_name="a project")

Your code.¶

```

Info

Every workspace member can use their own API key. If workspace is set it will be reported to that workspace. Remember, you can set your API Key in your config file. See Quick Start for more details.

Troubleshooting¶

Recall from above that you can set the configuration variable COMET_LOGGING_CONSOLE to "info" to see tracebacks for any Comet-based issues. This is often enough information to help track down a problem (for example, the reason why an image is not logged). However, if you need the maximum amount of debug information, you'll want to create a "Comet debug log file."

The first step in creating a "Comet debug log file" is to set two configuration variables: COMET_LOGGING_FILE and COMET_LOGGING_FILE_LEVEL as described above. For example, you can define the variables in the environment, or in your .comet.config file.

Here is how you can set them in the bash environment:

bash $ export COMET_LOGGING_FILE=/tmp/comet.log $ export COMET_LOGGING_FILE_LEVEL=debug

Here is the contents of a sample .comet.config file:

[comet_logging] file = /tmp/comet.log file_level = debug

Or, you can also define them at the same time as you run your script:

bash $ COMET_LOGGING_FILE_LEVEL=debug \ COMET_LOGGING_FILE=/tmp/comet.log \ python script.py

Finally, you can also put them right into the script itself before you import comet_ml:

```python import os os.environ["COMET_LOGGING_FILE"] = "/tmp/comet.log" os.environ["COMET_LOGGING_FILE_LEVEL"] = "debug"

import comet_ml ... ```

In these examples, the debugging logs will be in /tmp/comet.log, but you can put it where you like, and name it whatever you prefer. This log will show details on all of the steps of your experiment, and any details about failures. If you still have problems, please share this file with us via the Comet.ml Slack channel.

Another item to check is to make sure that your comet_ml version is up to date. You can find the latest version number on the Python Packaging comet_ml page. To upgrade, you can use the command:

bash $ pip install comet_ml --upgrade

In some cases, you may want to also update all of the packages that comet_ml depends on. You can do that with:

bash $ pip install comet_ml --upgrade --upgrade-strategy eager

System Check¶

If you are unable to successfully log data, or are on a new onprem installation and want to ensure that everything is set up correctly, you can run the Comet command-line script:

bash comet check

The comet check command will make sure that your system is properly configured for talking to the Comet.ml servers. More more information, see Comet Command-Line Utilities.

Running with Distributed and Parallel Processes¶

The Comet Python SDK can be used when the training process is distributed across several processes and potentially several GPUs and/or servers. For details, caveats, and examples, please see Using the Python SDK in distributed mode.