gdlib-config is missing, LgdImageCreateFromPng and LgdImageCreateFromPngPtr are missing

[viper7882]

Hi @ehrenbrav ,

I've attempted to make the DeepQNetwork package in both UBuntu 17.04 and Debian 9.0 and stuck during install_dependencies.sh script with the following errors:
Warning: variable CFLAGS was not passed in build_variables gcc -o gd.lo -cgdlib-config --features |sed -e "s/GD_/-DGD_/g"-O3 -Wall -fPIC -fomit-frame-pointergdlib-config --cflags pkg-config lua5.1 --cflags` -DVERSION="2.0.33r3" luagd.c
/bin/sh: 1: gdlib-config: not found
/bin/sh: 1: gdlib-config: not found
luagd.c:2171:33: error: ‘LgdImageCreateFromPng’ undeclared here (not in a function)
{ "createFromPng", LgdImageCreateFromPng },
^~~~~~~~~~~~~~~~~~~~~
luagd.c:2172:33: error: ‘LgdImageCreateFromPngPtr’ undeclared here (not in a function)
{ "createFromPngStr", LgdImageCreateFromPngPtr },
^~~~~~~~~~~~~~~~~~~~~~~~
Makefile:104: recipe for target 'gd.lo' failed
make: *** [gd.lo] Error 1

Error: Build error: Failed building.
Error. Exiting.
`

For Debian, I've checked where gdlib-config should be no longer available in the latest libgd-dev package, hence I've replaced it with pkg-config. I've managed to get rid of the gdlib-config error but still stuck at the subsequent error where LgdImageCreateFromPng and LgdImageCreateFromPngPtr are missing.

Do you have any idea what else I should have done?

[ehrenbrav]

Hmmmm...haven't seen that one before... I tried to reproduce on my machine but was unable to.

Two thoughts:

1. You're using /bin/sh instead of /bin/bash to execute the script, yes? I doubt that it would make a difference, but try using bash to execute the script.

2. The script should install libgd-dev (line 38), which places the required binary (on my machine) at /usr/bin/gdlib-config. I have version 2.1.0-5+deb8u9 of the libgd-dev package installed. Would you check two things:

• What version of libgd-dev you have installed.
• Whether you have the gdlib-config binary installed on your system (which gdlib-config)
• Check that your PATH is configured right (pretty unlikely this is the issue, but worth a try)

[viper7882]

Thank you for looking into this issue for me. I trust this is a generic issue which is not related to your program. Both of my Linux OSes installation are brand new, I doubt it is caused by any corruption from other programs.

Switching to bash does not seem to help:
neswrap installation completed Installing Lua-GD ... mkdir: cannot create directory ‘/home/ubuntu/installer/DeepQNetwork/torch/src’: File exists Cloning into 'lua-gd'... remote: Counting objects: 1418, done. remote: Total 1418 (delta 0), reused 0 (delta 0), pack-reused 1418 Receiving objects: 100% (1418/1418), 1.34 MiB | 806.00 KiB/s, done. Resolving deltas: 100% (1052/1052), done. Warning: variable CFLAGS was not passed in build_variables gcc -o gd.lo -c gdlib-config --features |sed -e "s/GD_/-DGD_/g"-O3 -Wall -fPIC -fomit-frame-pointergdlib-config --cflags pkg-config lua5.1 --cflags` -DVERSION="2.0.33r3" luagd.c
/bin/sh: 1: gdlib-config: not found
/bin/sh: 1: gdlib-config: not found
luagd.c:2171:33: error: ‘LgdImageCreateFromPng’ undeclared here (not in a function)
{ "createFromPng", LgdImageCreateFromPng },
^~~~~~~~~~~~~~~~~~~~~
luagd.c:2172:33: error: ‘LgdImageCreateFromPngPtr’ undeclared here (not in a function)
{ "createFromPngStr", LgdImageCreateFromPngPtr },
^~~~~~~~~~~~~~~~~~~~~~~~
Makefile:104: recipe for target 'gd.lo' failed
make: *** [gd.lo] Error 1

Error: Build error: Failed building.
Error. Exiting.`

I've googled where gdlib-config is part of libgd-dev. When I've performed sudo apt-get install libgd-dev and/or upgrade, the terminal simply returns the following:
Reading package lists... Done Building dependency tree Reading state information... Done libgd-dev is already the newest version (2.2.4-2). 0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

Output of which gdlib-config:
(None)

I've searched in the entire HDD using root account using sudo find / -name "gdlib-config", there is no sign of /usr/bin/gdlib-config or /usr/local/bin/gdlib-config either physical file or symbolic link.
Output of ls *-config in /usr/bin:
/usr/bin/apt-config /usr/bin/libpng-config /usr/bin/caca-config /usr/bin/ncurses5-config /usr/bin/freetype-config /usr/bin/pcre-config /usr/bin/i686-linux-gnu-pkg-config /usr/bin/pkg-config /usr/bin/icu-config /usr/bin/sdl-config /usr/bin/im-config /usr/bin/x86_64-linux-gnu-pkg-config /usr/bin/libpng16-config /usr/bin/x86_64-pc-linux-gnu-pkg-config

Online forum suggested to replace gdlib-config with pkg-config, it seems to work and as mentioned earlier, I could get rid of the missing gdlib-config temporary.

Output of echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

Without solving this dependency installation issue, I could not execute your fabulous Super Mario Bros Deep Q Network.

[ehrenbrav]

You know, I bet the problem is that you're using Debian Stretch. All my development and testing has been on Debian Jessie and I haven't gotten around to upgrading yet. Can you try installing on Debian Jessie?

[viper7882]

Hi @ehrenbrav ,

I've installed a fresh copy of Debian Jessie and sync down you files, the installation is completely successful. Thank you @ehrenbrav.

It also mean that all future users of Debian Stretch and Ubuntu 17.04 will encounter the same issue that I've reported earlier. There must be something got changed in these releases for gdlib.

[ehrenbrav]

Yes indeed - thanks for helping to uncover this issue!

[lhoggatt]

I'm running into the same problem as viper7882. Is there any way to run your incredible Deep Q Network on Ubuntu 17.04?

[ehrenbrav]

The short answer is...absolutely! The longer answer is, yes but I don't know how... It would require fixing the dependency issue, which is always a bit of a pain. If you'd be interested in tracking down the solution, I'd be grateful if you'd share your fix. Otherwise, if you're interested in the quick and dirty method of getting it working, I'd just run Debian Jessie in a VM :)

[lhoggatt]

It seems that gdlib-config has been depreciated and lua-gd has not been updated for about 4 years (thus, it does not support an alternative). I'm not really familiar enough with all of this to be of much more help unfortunately. I will use a VM if I want to run your network!

Thanks again for the quick reply and the help!

[viper7882]

Hi guys,

I've learned that the issue is related to ittner/lua-gd#8. After I've patched luagd.c file with the proposed fix in Debian 9 and the error message has disappeared. @ehrenbrav, you should consider to patch luagd.c file.

[everdom]

I had the same problem, and I had replaced gdlib-config with pkg-config, also patched luagd.c file, althogh I was able to compile, but all feathers were disabled. So I still couldn't run your incredible Super Mario..

$ make
gcc -o gd.lo -c pkg-config --features |sed -e "s/GD_/-DGD_/g" -O3 -Wall -fPIC -fomit-frame-pointer pkg-config --cflags pkg-config lua5.1 --cflags -DVERSION="2.0.33r3" luagd.c
Unknown option --features
Must specify package names on the command line
gcc -o gd.so gd.lo -shared pkg-config --ldflags pkg-config --libs -lgd
Unknown option --ldflags
Must specify package names on the command line
../../bin/luajit test_features.lua
Lua-GD version: lua-gd 2.0.33r3
Lua-GD features:
PNG support ..................... Disabled
GIF support ..................... Disabled
JPEG support .................... Disabled
XPM/XBM support ................. Disabled
FreeType support ................ Disabled
Fontconfig support .............. Disabled
Updating manifest for /media/everdom/000EC87C000FE6FF/Projects/AI/DeepQNetwork/torch/lib/luarocks/rocks
luagd 2.0.33r3-1 is now built and installed in /media/everdom/000EC87C000FE6FF/Projects/AI/DeepQNetwork/torch/ (license: MIT/X11)

Lua-GD installation completed

All done!

$ ./test_gpu.sh smb /home/everdom/TheBrainOfMario.t7
-framework neswrap -game_path /media/everdom/000EC87C000FE6FF/Projects/AI/DeepQNetwork/roms/ -name DQN3_0_1_smb_FULL_Y -env smb -env_params useRGB=true -agent NeuralQLearner -agent_params lr=0.01,ep=1,ep_end=0.1,ep_endt=500000,discount=0.99,hist_len=4,learn_start=5000,replay_memory=100,update_freq=4,n_replay=4,network="convnet_nes",preproc="net_downsample_2x_full_y",state_dim=7056,minibatch_size=256,rescale_r=1,ncols=1,bufferSize=1024,valid_size=1000,target_q=10000,clip_delta=1,min_reward=-10000,max_reward=10000 -actrep 8 -gpu 0 -random_starts 0 -pool_frms type="max",size=1 -seed 1 -threads 8 -network /home/everdom/TheBrainOfMario.t7 -gif_file ../gifs/smb.gif
Torch Threads: 8
Using GPU device id: 0
Torch Seed: 1
CUTorch Seed: 1791095845
Loading /media/everdom/000EC87C000FE6FF/Projects/AI/DeepQNetwork/roms/smb.zip...

PRG ROM: 2 x 16KiB
CHR ROM: 1 x 8KiB
ROM CRC32: 0x9a2db086
ROM MD5: 0xba39dde63ab209b1bc751e0535e72b18
Mapper #: 0
Mapper name: NROM
Mirroring: Vertical
Battery-backed: No
Trained: No

Power on
Initializing video... Video Mode: 256 x 224 x 32 bpp
Loading SDL sound with pulse driver...
Emulation speed 3200.0%
saveState NOT IMPLEMENTED.
loadState is not implemented...
Playing: smb
Loading the latest (not necessarily the best) model...
Set up Torch using these options:
game_path /media/everdom/000EC87C000FE6FF/Projects/AI/DeepQNetwork/roms/
csv_file
seed 1
name DQN3_0_1_smb_FULL_Y
gif_file ../gifs/smb.gif
verbose 2
network /home/everdom/TheBrainOfMario.t7
random_starts 0
gpu 1
env_params table: 0x41ebaed0
agent_params table: 0x41ad97d8
framework neswrap
agent NeuralQLearner
threads 8
actrep 8
tensorType torch.FloatTensor
env smb
gameOverPenalty 0
pool_frms table: 0x41ebaea8
qlua: test_agent.lua:65: attempt to call field 'createFromJpegStr' (a nil value)
stack traceback:
[C]: in function 'createFromJpegStr'
test_agent.lua:65: in main chunk

[everdom]

Hi guys, I've found a solution and run super mario successfully! haha
first I create a script install_lua_gd.sh at the project dir to manually download the source code of lua-gd:

#!/usr/bin/env bash
TOPDIR=$PWD

# Prefix:
PREFIX=$PWD/torch

echo "Installing Lua-GD ... "
mkdir $PREFIX/src
cd $PREFIX/src
#rm -rf lua-gd
git clone https://github.com/ittner/lua-gd.git
cd lua-gd
sed -i "s/LUABIN=lua5.1/LUABIN=..\/..\/bin\/luajit/" Makefile
$PREFIX/bin/luarocks make
RET=$?; if [ $RET -ne 0 ]; then echo "Error. Exiting."; exit $RET; fi
echo "Lua-GD installation completed"

echo
echo "All done!"

then I modified Makefile at /torch/src/lua-gd to replace gdlib-config with manual configurations:

# luagd -- gd bindings for the Lua Programming Language.
# (c) 2004-11 Alexandre Erwin Ittner <alexandre@ittner.com.br>
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
# IN NO EVENT SHALL THE AUTHOR OR COPYRIGHT HOLDER BE LIABLE FOR ANY
# CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#
# If you use this package in a product, an acknowledgment in the product
# documentation would be greatly appreciated (but it is not required).
#


# Lua-GD version. This one must be set.
VERSION=2.0.33r3

# Command used to run Lua code
LUABIN=../../bin/luajit

# Path to the utility 'gdlib-config'. This may be changed to compile the
# module with development versions of libgd.
GDLIBCONFIG=gdlib-config

# Optimization for the brave of heart ;)
OMITFP=-fomit-frame-pointer


# ---------------------------------------------------------------------------
# Automatic configuration using pkgconfig, gd-config and sed. These
# lines should work on most Linux/Unix systems. If your system does not
# have these programs you must comment out these lines and uncomment and
# change the next ones.

# Name of .pc file. "lua5.1" on Debian/Ubuntu
LUAPKG=lua5.1
OUTFILE=gd.so

CFLAGS=-O3 -Wall -fPIC $(OMITFP)
#CFLAGS+=`$(GDLIBCONFIG) --cflags` `pkg-config $(LUAPKG) --cflags`
CFLAGS+=`pkg-config $(LUAPKG) --cflags`
CFLAGS+=-DVERSION=\"$(VERSION)\"

#GDFEATURES=`$(GDLIBCONFIG) --features |sed -e "s/GD_/-DGD_/g"`
GDFEATURES=-DGD_XPM -DGD_JPEG -DGD_FONTCONFIG -DGD_FREETYPE -DGD_PNG -DGD_GIF
#LFLAGS=-shared `$(GDLIBCONFIG) --ldflags` `$(GDLIBCONFIG) --libs` -lgd
LFLAGS=-shared -lgd -lm $(OMITFP) 

INSTALL_PATH := `$(LUABIN) -e'                          \
    for dir in package.cpath:gmatch("(/[^?;]+)?") do    \
        io.write(dir)                                   \
        os.exit(0)                                      \
    end                                                 \
    os.exit(1)                                          \
'`



# ---------------------------------------------------------------------------
# Manual configuration for systems without pkgconfig.

#OUTFILE=gd.so
#CFLAGS=-O3 -Wall -fPIC $(OMITFP)
#CFLAGS+=`$(GDLIBCONFIG) --cflags` -I/usr/include/lua5.1
#CFLAGS+=-DVERSION=\"$(VERSION)\"
#GDFEATURES=`$(GDLIBCONFIG) --features |sed -e "s/GD_/-DGD_/g"`
#LFLAGS=-shared `$(GDLIBCONFIG) --ldflags` `$(GDLIBCONFIG) --libs` -lgd
#INSTALL_PATH=/usr/lib/lua/


# ---------------------------------------------------------------------------
# Manual configuration for Windows and systems without sed, pkgconfig, etc.
# Uncomment, change and good luck :)

#OUTFILE=gd.dll
#CFLAGS=-O3 -Wall -fPIC $(OMITFP)
#CFLAGS+=-IC:/lua5.1/
#CFLAGS+=-DVERSION=\"$(VERSION)\"
#GDFEATURES=-DGD_XPM -DGD_JPEG -DGD_FONTCONFIG -DGD_FREETYPE -DGD_PNG -DGD_GIF
#LFLAGS=-shared -lgd2 -lm $(OMITFP)
#INSTALL_PATH="C:/Program Files/lua/"
# ---------------------------------------------------------------------------


all: test

$(OUTFILE): gd.lo
	$(CC) -o $(OUTFILE) gd.lo $(LFLAGS)

test: $(OUTFILE)
	$(LUABIN) test_features.lua

gd.lo: luagd.c
	$(CC) -o gd.lo -c $(GDFEATURES) $(CFLAGS) luagd.c

install: $(OUTFILE)
	install -D -s $(OUTFILE) $(DESTDIR)/$(INSTALL_PATH)/$(OUTFILE)


# Rules for making a distribution tarball

TDIR=lua-gd-$(VERSION)
DFILES=COPYING README luagd.c lua-gd.spec Makefile test_features.lua
dist: $(DISTFILES)
	rm -f $(TDIR).tar.gz
	mkdir $(TDIR)
	mkdir -p $(TDIR)/doc $(TDIR)/demos $(TDIR)/debian
	cp $(DFILES) $(TDIR)
	cp demos/* $(TDIR)/demos/
	cp doc/* $(TDIR)/doc/
	cp debian/* $(TDIR)/debian/
	tar czf $(TDIR).tar.gz $(TDIR)
	rm -rf $(TDIR)

clean:
	rm -f $(OUTFILE) gd.lo
	rm -rf $(TDIR) $(TDIR).tar.gz
	rm -f demos/out.png demos/out.gif demos/counter.txt

.PHONY: all test install clean dist

then I run ./torch/src/lua-gd/make clean to clean all intermediate files
and finally I comment the line "git clone https://github.com/ittner/lua-gd.git" in install_lua_gd.sh to avoid downloading again and run ./install_lua_gd.sh to install again, then all features enabled

$ ./install_lua_gd.sh
Installing Lua-GD ...
Warning: variable CFLAGS was not passed in build_variables
gcc -o gd.lo -c -DGD_XPM -DGD_JPEG -DGD_FONTCONFIG -DGD_FREETYPE -DGD_PNG -DGD_GIF -O3 -Wall -fPIC -fomit-frame-pointer pkg-config lua5.1 --cflags -DVERSION="2.0.33r3" luagd.c
gcc -o gd.so gd.lo -shared -lgd -lm -fomit-frame-pointer
../../bin/luajit test_features.lua
Lua-GD version: lua-gd 2.0.33r3
Lua-GD features:
PNG support ..................... Enabled
GIF support ..................... Enabled
JPEG support .................... Enabled
XPM/XBM support ................. Enabled
FreeType support ................ Enabled
Fontconfig support .............. Enabled
Updating manifest for /media/everdom/000EC87C000FE6FF/Projects/AI/DeepQNetwork/torch/lib/luarocks/rocks
luagd 2.0.33r3-1 is now built and installed in /media/everdom/000EC87C000FE6FF/Projects/AI/DeepQNetwork/torch/ (license: MIT/X11)

Lua-GD installation completed

All done!