Skip to the content.

Welcome to the website for the learning tracks of the IPC 2023. The IPC learning tracks are a competition where algorithms learn domain-specific knowledge in an offline pre-processing phase and then feed that knowledge to a planner to solve new tasks from the same domain.

Results

Overview

The learning tracks use a similar setup as in 2008, 2011 and 2014. The main novelty in 2023 is that participants don’t have access to the PDDL domains and they don’t learn the domain knowledge themselves. Instead, they submit a fully automated learning system. Then the organizers learn the domain knowledge and evaluate the submitted planners on unseen test instances from the same domain. The main motivation for this setup is to make it easier to run learning systems from other authors, which increases reproducibility and helps to turn learning algorithms into off-the-shelve tools.

Participants will submit separate scripts for each submitted learner and planner:

We plan to pass between 50 and 100 tasks from “DOMAIN” in ascending “difficulty”. The learner will write its learned domain knowledge into the “DK” file. For details concerning the submission, see below.

We call a domain polynomial if all its tasks can be solved suboptimally in polynomial time. The competition will include both polynomial and non-polynomial domains.

Example domain knowledge includes but is not limited to:

For example PDDL tasks, see https://github.com/aibasel/downward-benchmarks.

Schedule

Event/Deadline Date
Call for domains July, 2022
Domain submission deadline December 9, 2022
Demo problems provided February, 2023
Team registration April 19, 2023
Feature stop (final submission) April 27, 2023
Planner abstract submission May 24, 2023
Contest run May - June, 2023
Results announced July 12, 2023
Results analysis due September 20, 2023

Environments

Single-Core

Multi-Core (canceled)

When the learner exceeds the time limit, we send them the SIGTERM signal, which can be caught to write the domain knowledge file and then gracefully exit. After an additional 60 seconds, we send SIGKILL.

Metrics

If an invalid plan is returned (or suboptimal plan for optimal metric), all tasks in the domain are counted as unsolved. If that happens in more than one domain, the entry is disqualified.

Satisficing

Agile

Optimal (cancelled)

PDDL Fragment

Learners and planners must support the following subset of PDDL 3.1: STRIPS, action costs, types, negative preconditions. Some of the training and testing tasks may be hard to ground.

Registration

This year, we will allow to submit multiple learners/planners to multiple tracks from a single repository. Thus, each team only needs one repository per code base and different parameters for different tracks can be set by providing multiple Apptainer files. More details to follow.

To register a team, the participants need to send an e-mail with a subject containing “[Registration for Learning Tracks]” to jendrik.seipp@liu.se. The email must contain:

  1. names of participants,
  2. email contacts,
  3. GitHub usernames,
  4. the number of repositories (code bases) the team needs (multiple learners/planners can be built from the same repository),
  5. a (tentative) list of environments and metrics, where the team intends to submit their learners/planners.

Based on that, we will create private repositories under the ipc2023-learning organization and add all participants as users with with write access and participants can commit to the repository as they wish until the “feature stop” deadline (see schedule).

After the feature stop deadline, we allow competitors to send only pull requests with bug fixes. We will review every pull request with its accompanying description of the bug fix to make sure that no big changes or parameter tuning are possible.

All participants must subscribe to the Google Group.

To propose a domain for the competition, please contact the organizers (see below).

Submission

The competitors must submit the source code of their learner and planners, which will be run by the organizers on the competition domains/problems, unknown to the competitors until this time. This way no fine-tuning of the learners or planners will be possible.

An important requirement for IPC 2023 competitors is to give the organizers the right to post their paper and the source code of their learners/planners on the official IPC 2023 web site, and the source code of all submissions must be released under a license allowing free non-commercial use.

As in the classical tracks of IPC 2018, we will use the container technology Apptainer (version 1.1.5, formerly known as Singularity) to promote reproducibility and simplify program compilation. In contrast to IPC 2018, we will host repositories of planners ourselves. The repositories will be hosted on GitHub under the ipc2023-learning organization, and they will be kept private until the end of the competition when we make them public.

We allow to submit multiple learners/planners to multiple tracks from a single repository. In each repository, we only consider the branch ipc2023-learning. Feel free to use other branches for development as you wish, but we will ignore them. Any pair of files called Apptainer.<shortname>.{learn|plan} in the root directory of this branch defines one entry. For the <shortname>, please use the name and variant of your planner as a short identifier (a single word, up to 16 characters long, starting with a letter, using only letters, digits, and underscores). If you build different versions of your planner from the same repository, use a different <shortname> per version. A single entry can participate in multiple tracks, see “Apptainer Images” for details.

Details for learners

Details for planners:

Apptainer images

We prepared a demo submission that showcases how to set up the repository and Apptainer scripts.

Your Apptainer recipe files have to specify the following labels:

Even though the labels will be the same in most cases for both Apptainer.<shortname>.{learn|plan} files, we ask you to add this information to both recipes.

To improve reproducibility, we require Apptainer images to be self-contained and licensed appropriately.

In addition to reproducibility and licensing issues, we ask that you make your image as small as possible using the following tricks:

Extended abstracts

All competitors must submit an up to 8-page paper describing their learner and planners (see schedule). After the competition we ask the participants to analyze the results of their learners and planners and submit an extended version of their paper.

Large files

GitHub repos have a file size limit of 100 MB. If you need files larger than this, you must upload them to an long-term file preservation site such as Zenodo and let your Apptainer script download them.

Bug-fixing Policy

To help us with the debugging process, planner authors will be responsible for detecting if the run of their planner and our analysis of the results was successful. After the feature stop deadline, we will run all planners on all tasks and give the participants access to the results of their planners. For each run, the data will contain the log files of the planner, measured time and memory consumption, exit code, and our conclusion about what this means in terms of solving the instance. We ask participants to check their results for any errors. If an error was caused by a bug in the planner, please send a pull request on GitHub with a detailed description of the bug and the fix. If the error was on our side (e.g., malformed PDDL) let us know as soon as possible. We will do at least two rounds of this starting after the “feature stop” deadline.

Participants

Baselines

Organizers

Public questions: ipc2023-learning@googlegroups.com

Contact organizers: jendrik.seipp@liu.se,javier/dot/segovia/at/upf/dot/edu

Acknowledgment: some of the text above has been adapted from the IPC 2023 classical track.