Hayato M. Yamanouchi

Tag: python

What Exactly Is YORU, a New Behavior Analysis Tool Powered by Object Detection?

In November 2024, a preprint was posted on bioRxiv describing YORU, a behavior analysis tool that takes a different approach from existing behavior analysis tools.
In this post, I would like to explain what kind of tool it is.

YORU preprint

YORU documentation

What is YORU?

YORU is a deep learning-based tool for analyzing animal behavior.

YORUのサイトより引用

Many deep learning-based tools, such as DeepLabCut and SLEAP, have already been released.
These tools use deep learning to track animal body parts and estimate posture (here I will call them tracking tools).

By estimating posture, they reveal what posture an animal is in and use this for behavior analysis.

YORU, by contrast, recognizes behavior using an object detection algorithm.
In short, it does not track body parts; instead, it defines behavior directly from the appearance of the behaving animal.

When we distinguish an apple from an orange, we rely on cues such as shape and color.
Object detection excels at classifying objects, so it makes exactly this kind of appearance-based distinction.

Applying this to animal behavior analysis means classifying animal behavior from its appearance.
Within YORU, this is referred to as a “behavior object.”

Whereas tracking tools represent behavior as points and lines, YORU analyzes behavior by enclosing it in a bounding box.

Advantages of defining behavior with object detection

The advantages of object detection-based behavior analysis include:

・Errors are less likely even as the number of individuals increases

・Less affected by the animal’s orientation

・Fast analysis speed

・Can capture behaviors that are hard to detect by tracking (such as a mouse crouching)

and so on.

It overcomes a weakness of tracking tools by making multi-animal behavior analysis simple and far less computationally costly.
It also makes it easy to analyze behaviors that are difficult to define from body part coordinates.

Consider, for example, the actions of opening and closing a hand.
With a tracking tool, you would track the fingertips with a camera and define the hand as open or closed, but if the fingertips become hidden when the hand closes, tracking fails and the positional information can no longer be computed accurately.

With object detection, however, the model looks at the overall shape of the open and closed hand, so it can still make an accurate call even when part of the hand is occluded.

As for robustness to increasing numbers of individuals: with a tracking tool, as the number of animals grows you must assign each body part to the correct individual in order to capture each animal’s behavior accurately, whereas YORU analyzes only the shape of each individual, so no identity assignment or part-to-individual matching is required.

Because analysis is also fast, real-time analysis is possible—you can trigger device control when a particular behavior occurs, enabling automation of behavioral experiments.

Disadvantages of defining behavior with object detection

There are, of course, drawbacks as well:

・Cannot capture a behavioral sequence

・Cannot identify individuals

・Cannot capture unknown behaviors

・Cannot capture behaviors that are not visually distinctive

・Provides no fine-grained detail about the defined behavior

These are precisely the strengths of tracking tools. Rather than one approach being better than the other, the two complement each other’s advantages and disadvantages.

Experimenters therefore need to choose the approach that best suits their situation.

What makes YORU remarkable?

Having covered the advantages and disadvantages of the object detection-based behavior analysis that underlies YORU, what exactly makes YORU itself stand out?

Everything runs through a GUI

YORU lets you perform behavior analysis without writing any code.

The same is true of tools such as DeepLabCut and SLEAP, and it goes a long way toward making these analyses accessible to biologists.

Design

Unlike previous tools, YORU’s design doesn’t really look like that of a research tool.

Built-in real-time analysis

Unusually for tools of this kind, YORU comes with a full GUI for real-time analysis.

Building a real-time analysis system is not that hard if you can program to some degree, but otherwise the barrier is quite high.

With YORU, however, you can carry out real-time analysis—and even fire external triggers—entirely without programming.

Adaptable to a wide range of experimental setups

The biggest hurdle in real-time analysis is figuring out how to combine the code that drives your own experimental apparatus with the analysis itself.

YORU adopts a plugin system that lets you choose the program used to control your external devices.

In other words, you simply select the plugin that matches your own system.

You can also write your own plugins: simply turning your device’s control program into a plugin makes it easy to link that hardware to YORU’s real-time analysis.

This makes for a remarkably user-friendly system that no other tool currently offers.

Closing thoughts

YORU has only just been released, so there are still rough edges in usability and clear gaps in the documentation.

That said, as these are addressed over time, I am very much looking forward to seeing how people put it to use.

2024-11-17
How to Install SLEAP [Updated February 2024]
Here I’d like to walk through how to install SLEAP, a machine learning-based tool for tracking animal body parts.

DeepLabCut is the best-known tracking tool, and I’ve written about how to install it myself in the past, while many others have posted articles about it as well.

SLEAP, however, has relatively few articles written about it in Japanese.
SLEAP has many advantages over DeepLabCut, so I decided to write up the installation procedure to make it a viable option for more people.

SLEAP’s documentation covers everything from installation to usage very clearly, complete with videos, so anyone comfortable with English should find it easy to get started.

The SLEAP paper is available here

The official SLEAP website is here

The SLEAP GitHub repository is here

Introduction

First, a note about the approach described in the official SLEAP installation guide:
```
mamba create -y -n sleap -c conda-forge -c nvidia -c sleap -c anaconda sleap=1.3.3
```
the one-line command that is supposed to handle environment creation, package installation, and everything else at once. I tried it on several computers, but in every case it hung indefinitely during the environment creation step and never completed.

So in this post I’ll introduce the alternative installation method that the developers also suggest.

Installation

We’ll basically follow the installation instructions on the official SLEAP site.

Environment

・Windows 11 Pro
・Verified on an NVIDIA 3080
・SLEAP 1.3.3
・Python 3.7.12

Downloading the files

First, download the files from the SLEAP GitHub repository to your computer.

If you’re using Git, run the following command.
```
git clone https://github.com/talmolab/sleap
```
If the git command isn’t available, either install git or download the files directly from GitHub.

Installing the GPU driver

Install the NVIDIA driver.
(Skip this step if it’s already installed.)

Installing Anaconda

Next, download and install the Windows 10 64-bit version from the Anaconda website.
Click Free Download on the Anaconda site and scroll down; you’ll see a screen like the one below.
Install the Windows installer on the far left.

Once Anaconda is installed, you should find “Anaconda Prompt” in your Windows app list.
We’ll use it to run the commands below to create the virtual environment, install the packages, and finally launch SLEAP.

Creating the virtual environment and installing

Next, run the following commands to create the virtual environment and install SLEAP.
The environment will be named “sleap”.
```
cd sleap
conda env create -f environment.yml -n sleap
```
Incidentally, the commands above work on computers with a GPU;
on machines without one, use the following instead.
```
cd sleap
conda env create -f environment_no_cuda.yml -n sleap
```
That completes the installation.

Checking that the installation worked

Activate the virtual environment and launch SLEAP with sleap-label.
```
conda activate sleap
sleap-label
```
You’ll need to activate the virtual environment (the conda activate sleap command) every time you reopen Anaconda Prompt.

After activating the environment, the prompt should change from (base) to (sleap).

If it launches, the installation was successful.

Checking version information for reporting in a paper

To check the version, activate the sleap environment and then run the following command.
```
python -c "import sleap; sleap.versions()"
```
This outputs:
```
SLEAP: 1.3.3
TensorFlow: 2.7.0
Numpy: 1.21.5
Python: 3.7.12
OS: Windows-10-10.0.22621-SP0
```
which shows the SLEAP version and related information.

To check whether SLEAP can use the GPU, run:
```
python -c "import sleap; sleap.system_summary()"
```
If the GPU is available, you’ll see output like the following.
```
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
       Initialized: False
     Memory growth: None
```
In future posts, I hope to walk through how to actually use SLEAP.

Bonus

For those who aren’t sure which computer to buy, I’ve started offering PC purchase consultations on Coconala!
あなたの要望に合わせてパソコンを選び、提案しますパソコン選びに困っている方々へ！様々な目的に対応できます！
I often pick out computers and give advice about them, and many friends have told me I could make money doing PC consultations.
That inspired me to give it a try!

I’ll recommend a machine that fits both what you want to use it for and your budget.
In particular, I’ve chosen and used many computers intended for machine learning.

And if you’d like, I can also advise you on what to look for the next time you buy a computer.

Computers I’ve picked out so far include lab analysis machines, everyday-use machines, game streaming machines, machines for incoming university students, machines that can run CAD for architecture students, and simple entry-level machines.

I use both Mac and Windows, so I can discuss and recommend either one!

Please feel free to make use of it.
2024-01-21
[Updated December 2022] How to Install YOLOX
What is YOLOX?

Among the many entries in the YOLO series, the latest one, YOLOX, was released in July 2021.

The YOLOX paper is available here (currently a preprint).
The GitHub repository is here.

YOLO is an object detection algorithm.
It is designed to run in real time and is extremely lightweight (inference alone can even run on a smartphone). It is also highly adaptable, and is widely used in IoT devices.

“Object detection algorithm” may sound complicated, but an easy analogy is a camera’s automatic face or eye focus.
A marker appears indicating “this is a face”; the same idea applies to other objects, and the algorithm can also classify what each one is.

There are many versions in the YOLO series: YOLO, YOLOv2, YOLOv3, YOLOv4, YOLOv5, and YOLOX.
It also runs not only on Python but on MATLAB as well, so you can adapt it to whichever platform you use.

Unlike the architecture used from YOLOv3 onward, YOLOX returns to the design of the original YOLO. It is reportedly faster and considerably more accurate.

Other blogs cover the details as well.

How to install YOLOX

Installation instructions are given on the GitHub page, but they are in English and do not cover using a virtual environment, so here I explain how to do it with an Anaconda virtual environment.

On both Windows and Mac, following the YOLOX manual exactly produced one error after another.
Errors will likely keep changing with future versions, so look them up as they come.

See below for how to use the conda command in the macOS Terminal.

Using Conda Commands on macOS

On Windows, I recommend using Anaconda Prompt.

Open the Terminal (Anaconda Prompt), and you will get a black window filled with text.
You should see (base) at the left of the last line.
This means you are not currently inside a virtual environment.

YOLOX will run without a virtual environment, but since environments can conflict with other packages, I recommend creating one.

First, enter the following code to create a virtual environment.
```
conda create -n YoloX python=3.8
```
YoloX is the name of the virtual environment; you can use any name you like.
You can also specify the Python version with python=3.8.

Then activate the virtual environment with the following code.
```
conda activate YoloX
```
If (base) on the left has changed to (YoloX), you are all set.

Installing CUDA

First, if this is your first time installing machine learning libraries and you plan to train YOLO on a GPU, install CUDA.
(Training on a CPU takes an absurdly long time, so I do not recommend it. Note that GPUs cannot currently be used on Apple Silicon machines.)
CUDA can be downloaded from the official site.

As for the CUDA version, I suggest checking which CUDA versions PyTorch supports and installing one of those.
PyTorch official site

To install an older version, click Download now and then, further down the page under Resources, use Archive of Previous CUDA Releases.

The latest version may work too, but for now, matching the version to PyTorch should let you run everything without trouble.

To check whether it is installed on Windows, open “Edit the system environment variables,” go to Advanced > Environment Variables, and see whether there is a path beginning with CUDA_PATH.
The number after the V indicates the version.

Installing PyTorch

Before installing, update pip.
Activate the virtual environment in Anaconda Prompt and run the following command.
```
python -m pip install --upgrade pip
```
Go to the PyTorch official site and choose the installation method that matches your environment.
One catch here is the Package field: be sure to select pip.
Installing with conda will cause errors later on.

Copy and paste the command under “Run this Command” to install it in your virtual environment.

Once the installation finishes, use the following command to check that PyTorch is installed.
```
pip list
```
As long as torch appears in the list, you’re good to go!!!
As a quick check, start python and run the following command.
```
import torch
print(torch.cuda.is_available())
```
If the output is True, the installation succeeded and torch is ready to run on the GPU.
(On a Mac or on a computer without a GPU it will return False, but as long as the import works you’re fine.)
If you get an error, go back and check the installation again.

Installing the packages required for YOLOX

Download (clone) the YOLOX files from GitHub.
If you can use Git commands, you can clone the repository with the code below.
```
git clone git@github.com:Megvii-BaseDetection/YOLOX.git
```
Next, move into the cloned folder and install the required packages.

First, edit the contents of requirements.txt in the cloned YOLOX folder as follows.
```
# TODO: Update with exact module version
numpy
#torch>=1.7
opencv_python
loguru
tqdm
#torchvision
thop
ninja
tabulate

# verified versions
# pycocotools corresponds to https://github.com/ppwwyyxx/cocoapi
#pycocotools>=2.0.2
#onnx==1.8.1
onnxruntime==1.8.0
onnx-simplifier==0.3.5
```
It seems that packages such as pycocotools cannot be installed with this command.

Once that’s done, save requirements.txt and run the command below in the terminal (Anaconda prompt).
```
cd YOLOX
pip3 install -r requirements.txt
pip3 install cython pycocotools
```
The downloaded YOLOX folder contains a file called “requirements.txt” that lists the required packages. The second command opens this file in read-only mode and installs the packages listed in it.
```
pip3 install cython pycocotools
```
This command installs cython and pycocotools.
It seems these have to be installed separately.

When you run this command, some of you may see the following error.
```
      building 'pycocotools._mask' extension
      error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
```
Microsoft C++ Build Tools is apparently required by pycocotools. (Oddly enough, the error also appears on Mac, even though there is no Mac version.)
Windows users should install it just to be safe.

After installing it, try the installation again with the code below.
```
pip3 install "git+https://github.com/philferriere/cocoapi.git#egg=pycocotools&subdirectory=PythonAPI"
```
If that still doesn’t work, install it with the following command.
```
conda install -c conda-forge pycocotools
```
In general we’ve been installing these packages with pip, but since this one simply won’t install that way, we resort to installing it with conda.

Next, download yolox_x.pth.
The download starts as soon as you click the link.

Move the downloaded file into the YOLOX folder you cloned from GitHub.
This gives you the model used for detection in the demo, placed wherever you like.

Add the following lines to the top of tools/demo.py so that the yolox folder can be accessed.
```
#demo.pyの中身に書き込みましょう。
import sys
import os
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
```
Without this, you’ll get an error saying the yolox package cannot be found.

Then run the following command in the terminal.
This lets you check whether the demo runs correctly.
```
python tools/demo.py image -n yolox-x -c yolox_x.pth --path assets/dog.jpg --conf 0.25 --nms 0.45 --tsize 640 --save_result --device gpu 
#gpuで動かさない場合には、gpuをcpuに書き換えてください。
```
Replace –device gpu with –device cpu as appropriate for your setup.

If the following image appears in YOLOX/YOLOX_outputs/yolox_x/vis_res/20…….., everything is working.

Bonus

For those who aren’t sure which computer to buy, I’ve started offering PC purchase consultations on Coconala!
あなたの要望に合わせてパソコンを選び、提案しますパソコン選びに困っている方々へ！様々な目的に対応できます！
I often pick out computers and give advice, and many friends told me I could make money doing PC consultations.
Inspired by that, I decided to give it a try!

I’ll recommend a machine that fits how you plan to use it and your budget.
In particular, I’ve chosen and used many computers for machine learning.

And if you’d like, I can also advise you on what to look for the next time you buy a computer.

Computers I’ve picked out so far include analysis machines for the lab, everyday-use machines, game-streaming machines, machines for incoming university students, machines that can run CAD for architecture students, and simple all-purpose machines.

I use both Mac and Windows, so I can talk about and recommend either!

Please feel free to make use of it.
2022-11-15
My elif conditional branching isn’t working!! [Python]
Have you ever wanted to branch on multiple conditions with an if statement, tried using elif, and found that although no error appears, the results somehow aren’t what you expected?
If so, the situation described below might be the cause.
For reference, since I made this mistake myself, I’m writing it down as a note.

The problematic program

The problem arises when you write two conditions in the if and elif statements and build a program that includes elif.
```
def trable(a, b):
    if a >= 10 & b >= 10:
        print("patern A")
    elif a >= 10 & b < 10:
        print("patern B")
    elif a < 10 & b >= 10:
        print("patern C")
    else:
        print("patern D")

trable(11, 11)
trable(11, 9)
trable(9, 11)
trable(9, 9)
```
When you run the program above, you might expect the cases to be sorted into patterns A, B, C, and D in order from the top, but the actual output is as follows.
```
patern A
patern B
patern C
patern B
```
What if we swap the order?
```
def trable(a, b):
    if a >= 10 & b >= 10:
        print("patern A")
    elif a < 10 & b < 10:
        print("patern B")
    elif a >= 10 & b >= 10:
        print("patern C")
    else:
        print("patern D")

trable(11, 11)
trable(9, 11)
trable(11, 9)
trable(9, 9)
```
If you swap the code for patterns B and C and write a program that you expect to output A, B, C, and D, the actual output is as follows.
```
patern A
patern D
patern D
patern D
```
elif itself is used as shown below when you want to define multiple conditions.
```
if 条件式A:
  条件式Aが真(True)となった場合の処理
elif 条件式B:
  条件式Aが偽(False)で、条件式Bが真(True)となった場合の処理
else:
  条件式Aが偽(False)で、条件式Bも偽(False)となった場合の処理
```
However, once the conditional expressions in the if and elif statements involve two conditions, the problem described above is likely to occur.
As a result, you don’t get the output you intended.

The tricky part is that no error is raised, so you can’t tell whether it worked until you actually look at the results.

How to fix it

When you want to branch into multiple cases using multiple conditions, avoid specifying multiple conditions in an elif statement.
```
def resolve(a, b):
    if a>= 10:
        if b >= 10:
            print("patern A")
        else:
            print("patern B")

    else:
        if b >= 10:
            print("patern C")
        else:
            print("patern D")

resolve(11, 11)
resolve(11, 9)
resolve(9, 11)
resolve(9, 9)
```
It becomes more cumbersome, but doing it this way produces exactly the output you expect.

Programming has unexpected pitfalls like this, so it’s a good lesson in carefully checking the code you write.

Addendum (August 2022)

It turns out the problem was not wrapping the conditions in parentheses.
```
def trable(a, b):
    if (a >= 10) & (b >= 10):
        print("patern A")
    elif (a < 10) & (b < 10):
        print("patern B")
    elif (a >= 10) & (b >= 10):
        print("patern C")
    else:
        print("patern D")
```
With this change, the branching worked correctly.
Alternatively, you can also solve it by writing and instead of &.
```
def trable(a, b):
    if a >= 10 and b >= 10:
        print("patern A")
    elif a < 10 and b < 10:
        print("patern B")
    elif a >= 10 and b >= 10:
        print("patern C")
    else:
        print("patern D")
```
& and and may seem the same, but & also acts as a bitwise AND, so in the original program
```
a >= 10 & b < 10
```
this apparently means a >= (10 & b) < 10, so with a=9 and b=9 the inequality becomes
9>=8<10, which evaluates to True.

It’s tricky, isn’t it…
2022-08-25
DeepLabCut: A Deep Learning-Based Behavior Analysis Tool

(The image above links to the paper.)

Behavioral analysis is a crucial experimental approach in biology, and accurate quantification of behavior is especially indispensable for understanding the brain.

Traditionally, high-accuracy pose estimation has been achieved by attaching markers (such as sensors) to the subject (a person or a laboratory animal).
However, sensors are a nuisance to the subject, and they can alter behavior or place constraints on it.

One marker-free alternative is to apply a skeleton model, but developing such a model is time-consuming and has to be done on a large scale.
There are also systems that estimate pose directly from images, but these generate enormous amounts of data, which again makes them very difficult for a single laboratory to run given equipment requirements.

With DeepLabCut, anyone with just a single computer can perform pose estimation, and do so very easily.
Using image recognition based on deep learning, it works marker-free and can recognize and track whatever body parts you choose.

Because the labeling is done by deep learning, you can analyze recorded videos without having to correct them beforehand.
In other words, tracking works even if the lighting across the field you want to analyze is uneven, or if the image is somewhat distorted depending on the viewing angle.

The page of the laboratory that developed DeepLabCut is here.
The GitHub repository is here.

From actually using DeepLabCut, I found that the computer used for analysis needs a GPU (graphics card) with at least 8 GB of memory.
Among gaming PC GPUs, that means a mid- to high-end card, so comfortable analysis is not possible on an everyday laptop.

That said, even without such a high-spec machine, you can use Google Colab to access a virtual GPU and run the analysis.
I haven’t tried it myself yet, though.

In actual analyses, I found the plotting accuracy to be very high.
However, it often fails when the background differs from that of the training data, so I felt it is best to prepare training data that closely matches the environment you actually want to analyze.

The paper showed that plotting accuracy improves when you label not only the few points you are interested in, but also the overall body—for example, a rough skeleton of the subject—even if those points are not used in the analysis.
Also, if the results are unsatisfactory after training and analysis, you can remove the problematic frames or add new training datasets, allowing you to actively improve the model.

DeepLabCut itself is very user-friendly and easy for anyone to use.
Why not give it a try in your own analyses?

I’d like to write about installing and using DeepLabCut in future posts.

2021-03-25

Hayato M. Yamanouchi

PhD student, Kamikouchi Laboratory, Nagoya University / JSPS Research Fellow (DC1)
Systems Ethology (Neuroethology × Informatics)

© 2026 Hayato M. Yamanouchi

On this site

Elsewhere

Contact details are on the About page.