1
0
mirror of https://github.com/gluster/glusterfs.git synced 2026-02-06 09:48:44 +01:00

84 Commits

Author SHA1 Message Date
Kaleb S. KEITHLEY
cef682c03d glusterfs: Build with Modern C, change set 5
GCC and Clang communities are hinting that in versions 14 and 16
respectively they will deprecate or disable legacy C89 features,
e.g. K&R1 style function definitions, among others.

In parallel, Fedora is going to start enforcing C99 as the minimum,
expected to land in F40. (I.e. around Spring 2024 IIRC.)

Currently Fedora is recommending that use of -Werror=implicit-int,
-Werror=implicit-function-declaration, -Werror=int-conversion,
-Werror=strict-prototypes, and -Werror=old-style-definition a set
of options to build with in the mean time to clean up code.

This change fixes a subset of the errors found when compiling with
this set of options.

Change-Id: I2025e39566aeb1f68836226d6d18c43418ca7f95
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2022-11-02 10:05:46 +01:00
Yaniv Kaul
83ded619e7 Libvirt xml changes to gf_test (#3849)
* gftest: domain XML changes

* Reduce the no. of packages installed and repos used in 'yum' command
2022-10-06 08:03:01 +02:00
Xavi Hernandez
29a7be3d3f gftest: fix commit checkout
The specified commit in 'gftest build' command was interpreted as a
branch name instead of a commit hash, which was incorrect and caused
failures when trying to use a it.

Fixes: #3847
Change-Id: I3ae3f255996cced722573fdc9ae189acfb22eb85
Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
2022-10-05 09:51:20 +02:00
Xavi Hernandez
1278939a03 tests: new testing tool
A new tool gftest has been created to automate and simplify the
preparation of an environment to run tests. It can also process
the full set of regression tests in parallel to significantly
reduce the total execution time.

It's written in python and requires these components:

- libvirt
- qemu-kvm
- gnupg
- python-click

All other modules should already be present in a default installation.

Using this tool is quite simple:

1. Create the VM

```
gftest create [OPTIONS] <name>

Options:
  --template <name> Template to use (default: centos7)
  --port <num>      Port to access the VM (default: 2222)
  --key <path>      Private key for SSH (default: ~/.ssh/id_rsa)
  --cpus <num>      Number of virtual cores of the VM
  --memory <num>    Memory assigned to the VM in GiB
  --disk <num>      Disk space for the VM in GiB
```

This creates a KVM VM named <name> that can be accessed through the
specified port on the local host (i.e. `ssh -p <port> root@127.0.0.1`)
using the specified private key. All other option will be taken from
the template if not present.

This process also creates a container inside the VM with all the
required packages and configurations to build Gluster and run the
tests.

2. Build the code

```
gftest build [OPTIONS] <name>

Options:
  --commit <id>  SHA/name of the commit to compile
```

This copies the current git repo to the VM and compiles the specified
commit in a container image.

3. Start the workers

```
gftest spawn [OPTIONS] <name>

Options:
  --workers <num>  Number of workers to create
  --space <num>    Space allocated for running tests in GiB
```

This starts the specified number of container instances of the compiled
Gluster image.

4. Run the tests

```
gftest run <name> <output dir>
```

This runs the full set of tests in parallel in all available containers
and puts the results into the <output dir>.

It also collects statistics for each test and CPU and memory state
during the full run.

5. Access the VM or container for manual testing/debugging

```
gftest sh <name> [<idx>]
```

Without `idx` it opens a shell session to the VM. With an index it opens
a shell session to the idx-th testing container.

There are some additional commands:

- `gftest kill <name>`
  Stops and destroys all testing containers.

- `gftest shutdown <name>`
  Gracefully shuts down the VM.

- `gftest poweroff <name>`
  Forcibly stops the VM.

- `gftest poweron <name>`
  Starts the VM.

Updates: #3469
Change-Id: I169877a4c5197d001bf2822ad7116559f7d78754
Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
2022-09-30 12:51:17 +02:00
Ravishankar N
146b35d1ba extras, tools, doc: remove offensive language
Replace master and slave terminology in geo-replication with primary and
secondary respectively.

Change-Id: I3eb9242d2ce8340435265b764d28221d50f872c8
Signed-off-by: Ravishankar N <ravishankar@redhat.com>
2020-12-30 15:55:22 +05:30
Yaniv Kaul
081c71d7f2 glusterd-utils.c: display which options have changed
Display which options were not changed from the default.
The user may have opted to change some global or volume options
from the default they were initially. Display '(DEFAULT)' if the values
used are those that were not explicitly set by the user.

Example output:
Option                                  Value
------                                  -----
cluster.server-quorum-ratio             50
cluster.enable-shared-storage           disable (DEFAULT)
cluster.op-version                      80000
cluster.max-op-version                  90000
cluster.brick-multiplex                 disable (DEFAULT)
cluster.max-bricks-per-process          250 (DEFAULT)
glusterd.vol_count_per_thread           100 (DEFAULT)
cluster.localtime-logging               disable (DEFAULT)
cluster.daemon-log-level                INFO (DEFAULT)

Since glusterfind uses the value, it is now filtering the value
and only picking the 1st word (which is the value itself) and ignores
the rest, which may now be '(DEFAULT)'.

Fixes: #1357
Change-Id: I7c59055158d099a5de38943f2169fd02c77f5d09
Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
2020-07-07 21:49:50 +03:00
Emmanuel Dreyfus
b6a657e8eb NetBSD build fixes
- Make sure -largp is used at link time
- PTHREAD_MUTEX_ADAPTIVE_NP is not available, use PTHREAD_MUTEX_DEFAULT instead
- Avoid non POSIX [[ ]] in scripts
- Do not check of lock.spinlock is NULL since it is not a pointer
  (it is not a pointer on Linux either)

Change-Id: I5e04a7c552d24f8a473c2b837828d1bddfa7e128
Fixes: #1347
Type: Bug
Signed-off-by: Emmanuel Dreyfus <manu@netbsd.org>
2020-06-30 16:42:36 +02:00
Shwetha K Acharya
fb45a0639e tools/glusterfind: Remove an extra argument
fixes: bz#1790748
Change-Id: I1cb12c975142794139456d0f8e99fbdbb03c53a1
Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>
2020-01-14 10:51:06 +05:30
Sunny Kumar
fe0fd5241a glusterfind: python3 compatibility
Problem:
While we delete gluster volume the hook script 'S57glusterfind-delete-post.py'
is failed to execute and error message can be observed in glusterd log.

Traceback:
  File "/var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post", line 69, in <module>
    main()
  File "/var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post", line 39, in main
    glusterfind_dir = os.path.join(get_glusterd_workdir(), "glusterfind")
  File "/usr/lib64/python3.7/posixpath.py", line 94, in join
    genericpath._check_arg_types('join', a, *p)
  File "/usr/lib64/python3.7/genericpath.py", line 155, in _check_arg_types
    raise TypeError("Can't mix strings and bytes in path components") from None
TypeError: Can't mix strings and bytes in path components

Solution:

Added the 'universal_newlines' flag to Popen to support backward compatibility.

Change-Id: Ie5655b11b55535c5ad2338108d0448e6fdaacf4f
Fixes: bz#1789478
Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
2020-01-10 14:28:35 +00:00
Kotresh HR
5fb713fde7 glusterfind: Fix py2/py3 issues
1. In dictionary values(), returns list in py2 and not in py3.
   So explicitly convert it into list.
2. xattr module returns values in bytes. So explicitly convert
   them to str to work both with py2 and py3

fixes: bz#1789439
Change-Id: I27a639cda4f7a4ece9744a97c3d16e247906bd94
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2020-01-10 16:48:14 +05:30
Milind Changire
994f4f8f31 tools/glusterfind: handle offline bricks
Problem:
glusterfind is unable to copy remote output file to local node when a
remove-brick is in progress on the remote node. After copying remote
files, in the --full output listing path, a "sort -u" command is run on
the collected files. However, "sort" exits with an error code if it
finds any file missing.

Solution:
Maintain a map of (pid, output file) when the node commands are started
and remove the mapping for the pid for which the command returns an
error. Use the list of files present in the map for the "sort" command.

Change-Id: Ie6e019037379f4cb163f24b1c65eb382efc2fb3b
fixes: bz#1410439
Signed-off-by: Milind Changire <mchangir@redhat.com>
Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>
2017-01-05 19:53:19 +05:30
Shwetha K Acharya
d29f4779bd geo-rep: Fix spelling errors
Fixes: bz#1741779
Change-Id: I708b6b7e6c520dee10445528e6f99ba38e141c25
Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>
2019-08-16 11:37:40 +05:30
Milind Changire
17c24b20ea glusterfind: integrate with gfid2path
Integration with gfid2path helps avoid file-system crawl and saves
precious time. Extended attributes starting with "trusted.gfid2path."
are read and the <PGFID>/<BN> values are extracted and the <PGFID> is
iteratively resolved from the brick backend to arrive at the full path.

Change-Id: I593b02880e3413b77bfceed4a36b00d401f03bc0
fixes: #529
Signed-off-by: Milind Changire <mchangir@redhat.com>
Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>
2019-02-19 12:49:12 +05:30
Saravanakumar Arumugam
5a21722eb4 tools/glusterfind : validate session name
Validate a session name(during create) for the following:
1. minimum 2 character length.
2. Maximum 256 characters.
3. No special characters apart from underscore, hyphen allowed.

Also, validate volume(expect, while using glusterfind list).

Change-Id: I1b1e64e218f93d0a531d3cf69fc2ce7e2ed11d01
BUG: 1241494
Signed-off-by: Saravanakumar Arumugam <sarumuga@redhat.com>
2015-07-09 15:56:28 +05:30
Sunny Kumar
dc37134990 libgfchangelog : use find_library to locate shared library
Issue:

libgfchangelog.so: cannot open shared object file

Due to hardcoded shared library name runtime loader looks for particular version of
a shared library.

Solution:

Using find_library to locate shared library at runtime solves this issue.

Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 323, in main
    func(args)
  File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 82, in subcmd_worker
    local.service_loop(remote)
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1261, in service_loop
    changelog_agent.init()
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__
    return self.ins(self.meth, *a)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__
    raise res
OSError: libgfchangelog.so: cannot open shared object file: No such file or directory

Change-Id: I3dd013d701ed1cd99ba7ef20d1898f343e1db8f5
fixes: bz#1699394
Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
2019-04-12 19:55:10 +05:30
Kaleb S. KEITHLEY
1f874f4220 glusterfind: revert shebangs to #!/usr/bin/python3
Using #!/usr/bin/env python is illegal in Fedora and RHEL packaging.
(Debian and SUSE packaging will complain about it too but don't
enforce it like Fedora packaging does. Yet.)

We went through a giant exercise fixing these once already.

The python needs to be python2/python3 clean. There is a build
scriptlet that converts the shebangs back to /usr/bin/python on rhel7.

Change-Id: If208a2557f7e5d727cda98c27905e249cf485d5b
updates: bz#1193929
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2019-02-22 09:25:26 -05:00
Shwetha K Acharya
e94abcf65c glusterfind: python2 to python3 compat
Made necessary modifications to ensure python3 compatibilty.

fixes: bz#1658116
Change-Id: I5cf1d0447eaf3c44eb444245d1f67aadd60705c3
Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>
2018-12-10 13:45:34 +05:30
Shwetha K Acharya
c5bcb98331 tools/glusterfind: option to display only files/directories
Updated full find to filter for files and directories.
--full --type f  lists only the files,
--full --type d lists only the directories,
--full (by default) lists both files and directories.

fixes: #579
Change-Id: If2c91a21a131667d5de34635c1846013e8fa20b7
Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>
2018-12-31 10:55:21 +05:30
Harpreet Kaur
dab1a5e7d3 tools: stack-buffer-overflow reported by asan
This patch fixes buffer overflow in $SRC/tools/setgfid2path/src/main.c
Memory access at offset 196 overflows "pgfid" variable
SUMMARY: AddressSanitizer: stack-buffer-overflow (/lib64/libasan.so.5+0x4e935)

updates: bz#1633930

Change-Id: Ib508c57e96c46b628f63c511437b853b39ae7955
Signed-off-by: Harpreet Kaur <hlalwani@redhat.com>
2018-12-11 15:07:56 +05:30
ShyamsundarR
c9a3f1e267 libglusterfs: Move devel headers under glusterfs directory
libglusterfs devel package headers are referenced in code using
include semantics for a program, this while it works can be better
especially when dealing with out of tree xlator builds or in
general out of tree devel package usage.

Towards this, the following changes are done,
- moved all devel headers under a glusterfs directory
- Included these headers using system header notation <> in all
code outside of libglusterfs
- Included these headers using own program notation "" within
libglusterfs

This change although big, is just moving around the headers and
making it correct when including these headers from other sources.

This helps us correctly include libglusterfs includes without
namespace conflicts.

Change-Id: Id2a98854e671a7ee5d73be44da5ba1a74252423b
Updates: bz#1193929
Signed-off-by: ShyamsundarR <srangana@redhat.com>
2018-11-29 14:08:06 -05:00
Milind Changire
c9c0a48eae glusterfind: add logs to identify parsing phases
Add logs to idenitfy start and finish of changelog parsing phases.

Change-Id: Id250231f2af7829f887401d30ac98875ae1ae793
fixes: bz#1632236
Signed-off-by: Milind Changire <mchangir@redhat.com>
2018-09-24 17:53:35 +05:30
Gluster Ant
968d41ae2b Land part 2 of clang-format changes
Change-Id: Ia84cc24c8924e6d22d02ac15f611c10e26db99b4
Signed-off-by: Nigel Babu <nigelb@redhat.com>
2018-09-12 17:52:45 +05:30
Kotresh HR
734dfb10f1 multiple: Fix python2 to python3 compat
Change-Id: Iac241166d7a35dc7cc6cf07850f9f1bce38fe207
Updates: #411
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2018-09-05 04:30:35 -04:00
Kaleb S. KEITHLEY
e7ff038968 core: python3
see https://review.gluster.org/#/c/19788/,
    https://review.gluster.org/#/c/19871/,
    https://review.gluster.org/#/c/19952/,
    https://review.gluster.org/#/c/20104/,
    https://review.gluster.org/#/c/20162/,
    https://review.gluster.org/#/c/20185/,
    https://review.gluster.org/#/c/20207/,
    https://review.gluster.org/#/c/20227/,
    https://review.gluster.org/#/c/20307/,
    https://review.gluster.org/#/c/20320/,
    https://review.gluster.org/#/c/20332/,
    https://review.gluster.org/#/c/20364/,
    https://review.gluster.org/#/c/20441/, and
    https://review.gluster.org/#/c/20484

shebangs changed from /usr/bin/python2 to /usr/bin/python3.
(Reminder, various distribution packaging guidelines require use
of explicit python version and don't allow '#!/usr/bin/env python',
regardless of how handy that idiom may be.)

glusterfs.spec(.in) package python{2,3}-gluster and python2 or
python3 dependencies as appropriate.

configure(.ac):
+ test for and use python2 or python3 as appropriate. If build
  machine has python2 and python3, use python3. Override by
  setting PYTHON=/usr/bin/python2 when running configure.
+ PYTHONDEV_CPPFLAGS from python[23]-config --includes is a
  better match to the original python sysconfig.get_python_inc().
  All those other extraneous flags breaks the build.
+ Only change the shebangs once. Changing them over and over
  again, e.g., during a `make glusterrpms` in extras/LinuxRPM
  just sends make (is it really make that's looping?) into an
  infinite loop. If you figure out why, let me know.
+ Oldest python2 is python2.6 on CentOS 6 and Debian 8 (Jessie).
  Everything else has 2.7 or 3.x
+ logic from https://review.gluster.org/c/glusterfs/+/21050, which
  needs to be removed/merged after that patch is merged.

Builds on CentOS 6, CentOS 7, Fedora 28, Fedora rawhide, and the
mysterious RHEL > 7.

Change-Id: Idae21d3b6f58b32372e1daa0d234e491e563198f
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-08-29 11:09:27 -04:00
Yaniv Kaul
4e7093dfc8 multiple files: move from strlen() to sizeof()
{glusterfsd|glusterfsd-mgmt|quota-common-utils|xlator|tier|stripe}.c
tools/setgfid2path/src/main.c
xlators/cluster/afr/src/afr-inode-read.c
{glusterfs-acl|glusterfs}.h

For const strings, just do compile time size calc instead of runtime.

Compile-tested only!

Change-Id: I303684b1ff29b05c10126fb1057f507e404ced07
updates: bz#1193929
Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
2018-08-21 19:18:17 +03:00
Kotresh HR
fc27fc4ed8 glusterfind: Fix problems in python2 -> python3 compat
1. Import configparser module correctly
2. Fix issues with urllib usage

Updates: #411
Change-Id: I060af29f1ebac3a5e2aeeee2720b7372d4c31697
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2018-08-22 02:05:59 -04:00
Yaniv Kaul
7dd8b0a2df All: run codespell on the code and fix issues.
Please review, it's not always just the comments that were fixed.
I've had to revert of course all calls to creat() that were changed
to create() ...

Only compile-tested!

Change-Id: I7d02e82d9766e272a7fd9cc68e51901d69e5aab5
updates: bz#1193929
Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
2018-07-16 17:03:17 +03:00
Kaleb S. KEITHLEY
df99864878 core/various: python3 compat, prepare for python2 -> python3
see https://review.gluster.org/#/c/19788/,
    https://review.gluster.org/#/c/19871/, and
    https://review.gluster.org/#/c/19952/

This patch adds version agnostic imports for urllib, cpickle,
socketserver, _thread, queue, etc., suggested by Aravinda in
https://review.gluster.org/#/c/19767/1

Note: Fedora packaging guidelines require explicit shebangs, so
popular practices like #!/usr/bin/env python and #!/usr/bin/python
are not allowed; they must be #!/usr/bin/python2 or #!/usr/bin/python3

Note: Selected small fixes from 2to3 utility. Specifically apply,
basestring, funcattrs, idioms, numliterals, set_literal, types, urllib,
and zip have already been applied.

Note: these 2to3 fixes report no changes are necessary: exec, execfile,
exitfunc, filter, getcwdu, intern, itertools, metaclass, methodattrs, ne,
next, nonzero, operator, paren, raw_input, reduce, reload, renames, repr,
standarderror, sys_exc, throw, tuple_params, xreadlines.

Change-Id: I8d393064a1837874d8b4bc87c8ce05c679664642
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-05-30 08:15:29 -04:00
Kaleb S. KEITHLEY
84cda6024a core/various: python3 compat, prepare for python2 -> python3
see https://review.gluster.org/#/c/19788/ and
    https://review.gluster.org/#/c/19871/

Selected small fixes from 2to3 utility. Specifically apply, basestring,
funcattrs, idioms, numliterals, set_literal, types, urllib, zip

Note: these 2to3 fixes report no changes are necessary: exec, execfile,
exitfunc, filter, getcwdu, intern, itertools, metaclass, methodattrs, ne,
next, nonzero, operator, paren, raw_input, reduce, reload, renames, repr,
standarderror, sys_exc, throw, tuple_params, xreadlines.

Any 2to3 fixes not in the above two lists have more extensive changes
which will follow in separate patches.

most unicode changes suggested by 2to3 will need to be applied at the
same time as changing the shebangs from python2 to python3. Prashanth
notes that unicode strings in py2 need 'u' prefix; unicode strings in
py3 3.0, 3.1, and 3.2 a 'u' prefix will throw an error, but in py3 3.3+
it is legal (or just ignored). All Linux dists we care about have 3.3
or later so we can leave 'u' prefixes on unicode strings.

Change-Id: I49bba2f328b0ee24b9a8115a7183be979981563e
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-05-02 08:48:32 -04:00
Kaleb S. KEITHLEY
7a8418e31f core/build/various: python3 compat, prepare for python2 -> python3
Note 1) we're not supposed to be using #!/usr/bin/env python, see
https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#Shebang_lines

Note 2) we're also not supposed to be using "!/usr/bin/python,
see https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build#Quick_Opt-Out

The previous patch (https://review.gluster.org/19767) tried to do too
much in one patch, so it was abandoned.

This patch does two things:
1) minor cleanup of configure(.ac) to explicitly use python2
2) change all the shebang lines to #!/usr/bin/python2 and add them
where they were missing based on warnings emitted during rpmbuild.

In a follow-up patch python2 will eventually be changed to python3.

Before that python2-isms (e.g. print, string.join(), etc.) need to be
converted to python3. Some of those can be rewritten in version agnostic
python. E.g. print statements become print() with "from __future_ import
print_function". The python 2to3 utility will be used for some of those.
Also Aravinda has given guidance in the comments to the first patch for
changes.

updates: #411
Change-Id: I471730962b2526022115a1fc33629fb078b74338
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-03-28 10:14:39 -04:00
Niklas Hambüchen
83b4c47a32 glusterfind: Log remote stderr on node_cmd error. Fixes #1559130
The problem of lost stderr was introduced in
commit feea851fad.

Change-Id: Ic98f9bc9682ae3bd9c3ebea3855667fc8ba2843d
BUG: 1559130
Signed-off-by: Niklas Hambüchen <mail@nh2.me>
2017-09-08 02:40:01 +02:00
Niklas Hambüchen
f73181612f python: Remove all uses of find_library. Fixes #1450593
`find_library()` doesn't consider LD_LIBRARY_PATH on Python < 3.6.

Change-Id: Iee26085cb5d14061001f19f032c2664d69a378a8
BUG: 1450593
Signed-off-by: Niklas Hambüchen <mail@nh2.me>
2017-05-13 18:54:36 +02:00
Niklas Hambüchen
629bce68a1 glusterfind: Show C function when raising ChangelogException
Fixes: #432

Change-Id: I9ab031e098aff717e619d9deb6410281b96de14a
Signed-off-by: Niklas Hambüchen <mail@nh2.me>
2017-12-31 17:46:40 +01:00
Niels de Vos
21595abf2b build: add --without-server option
With Gluster 4.0 we will not provide the server components for EL6 and
older. At one point Gluster 4.x will get GlusterD2, which requires
Golang tools in the distribution. EL6 does not contain these at the
moment.

With this change, it is possible to `./configure --without-server` which
prevents building glusterd and the xlators for the bricks. Building RPMs
can pass `--without server` and the glusterfs-server sub-package will
not be created.

Change-Id: I97f5ccf9f2c76e60d9af83915fc59fae57ad6d25
BUG: 1074947
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2018-01-31 16:38:34 +01:00
Niklas Hambüchen
e6acec68a7 glusterfind: Speed up gfid lookup 100x by using an SQL index
Fixes #1529883.

This fixes some bits of `glusterfind`'s horrible performance,
making it 100x faster.

Until now, glusterfind was, for each line in each CHANGELOG.* file,
linearly reading the entire contents of the sqlite database in
4096-bytes-sized pread64() syscalls when executing the

  SELECT COUNT(1) FROM %s WHERE 1=1 AND gfid = ?

query through the code path:

  get_changes()
    parse_changelog_to_db()
      when_data_meta()
        gfidpath_exists()
          _exists()

In a quick benchmark on my laptop, doing one such `SELECT` query
took ~75ms on a 10MB-sized sqlite DB, while doing the same query
with an index took < 1ms.

Change-Id: I8e7fe60f1f45a06c102f56b54d2ead9e0377794e
BUG: 1529883
Signed-off-by: Niklas Hambüchen <mail@nh2.me>
2017-12-29 15:49:13 +01:00
Aravinda VK
3cf87a175d tools/setgfid2path: Tool to set GFID to Path xattr in brick backend
Once storage/gfid2path feature is enabled using `gluster volume set
<volname> storage.gfid2path enable`, it starts recording the gfid2path
xattr on each files. But this feature will not add xattr to the existing
files.

This tool accepts the file path as argument and sets the necessary xattr
required for this feature.

Change-Id: I75ad82c86ce482950645e687ff2e33b413fa53da
Updates: #139
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: https://review.gluster.org/17839
Smoke: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Tested-by: Kotresh HR <khiremat@redhat.com>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
2017-07-20 17:08:12 +05:30
Aravinda VK
4bdf92b63b tools/glusterfind: Fix encoding to encode only space,newline and percent chars
libgfchangelog was encoding path using spec rfc3986, but encoding only
required for SPACE, NEWLINE and PERCENT chars since the NEWLINE char is
used as record separator and SPACE as field separator in the parsed
changelogs output.

Changed the encoding function to encode only SPACE, NEWLINE and PERCENT chars

BUG: 1451724
Change-Id: Ic1dea824d23493dedcf3db45f353f90572f4e046
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: https://review.gluster.org/17788
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Milind Changire <mchangir@redhat.com>
2017-07-03 14:51:21 +05:30
Milind Changire
040b7c0ac4 tools/glusterfind: unquote DELETE path before further processing
Problem:
DELETE path is quoted before it reaches glusterfind. This wasn't handled
in the glusterfind code leading to double quoting of path separator
'%2F' to '%252F' i.e. the '%' character in '%2F' itself was quoted to
'%25'

Solution:
unquote the the deleted path before further processing

Change-Id: I2dfbbd7792dc0f9da5c8e02093b0f1c031ff344a
BUG: 1465024
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: https://review.gluster.org/17629
Smoke: Gluster Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2017-06-26 18:20:23 +05:30
Milind Changire
eb31f156eb tools/glusterfind: initialize variable 'end'
Problem:
syntax error due to uninitialized variable 'end'

Solution:
set end = -1

Change-Id: Ia8b500e9dd5304d0832e5f594abca0b5b2ab357e
BUG: 1462241
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: https://review.gluster.org/17557
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2017-06-16 18:39:08 +05:30
Milind Changire
e762a2f212 tools/glusterfind: add --end-time option
Add optional --end-time argument to be used with --since-time
when using the "query" command.

"start" and "end" times are passed in the command-line to
changelog.py only if --full has not been specified on command-line.
-1 is passed to changelog.py as end time if user has not supplied
--end-time on command-line.

brickfind.py:
Remove unused "start" command-line argument.

Also:
Minor indentation changes to keep flake8-2.7 happy.

Change-Id: I063ef5459916f711503881ade5c4fc32374edad0
BUG: 1453151
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: https://review.gluster.org/17439
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2017-06-06 12:27:28 +05:30
Milind Changire
51763ff6bd tools/glusterfind: add --field-separator option
Problem:
Default field separator is a space character.
This gets in the way if the file name itself has embedded spaces.

Solution:
Add --field-separator option to "pre" and "query" commands.
The field separator string will be used to separate strings in the
output lines that get written to the output file.

eg.
old output:
NEW file1.txt
RENAME file2 Copy.txt file3.txt

with --field-separator as "==="
new output:
NEW===file1.txt
RENAME===file2 Copy.txt===file3.txt

Change-Id: I71e878fed58ba1113d97044ac9f6404ee66227c7
BUG: 1453151
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: https://review.gluster.org/17481
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2017-06-06 23:43:54 +05:30
Kaleb S. KEITHLEY
583ded3058 build/packaging: Debian and Ubuntu don't have /usr/libexec
GLUSTERFS_LIBEXECDIR is effectively hard-coded to /usr/libexec/glusterfs
in configure(.ac)

Debian-based distributions don't have a /usr/libexec/ directory

This issues is partially mitigated by the use of $libexecdir in
some of the Makefile.am files, but even so the incorrectly defined
GLUSTERFS_LIBEXECDIR results in various things such as gsyncd,
glusterfind, eventsd, etc., trying to invoke other scripts and
programs from a location that doesn't exist.

And once we correctly define GLUSTERFS_LIBEXECDIR, then we might as
well use it appropriatedly.

Change-Id: If5219cadc51ae316f7ba2e2831d739235c77902d
BUG: 1430841
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
Reviewed-on: https://review.gluster.org/16880
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Joe Julian <me@joejulian.name>
Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
2017-03-09 12:18:40 -05:00
Milind Changire
9a2f4f48dd tools/glusterfind: delete temporary folder
Problem:
Run specific temporary folder created under
/usr/var/lib/misc/glusterfsd/glusterfind/<session>/<volume>/
remains undeleted.

Solution:
Delete the temporary folder from all nodes.

Change-Id: I0edaf868aebb01b15c489434bbb26fe853351384
BUG: 1413526
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: https://review.gluster.org/16416
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2017-01-16 15:27:09 +05:30
Milind Changire
5bf1ba5bc5 tools/glusterfind: avoid deleting keys directory
Problem:
gluster volume delete mistakenly deletes the .keys directory under
/var/lib/glusterd/glusterfind.

Solution:
Check for ".keys" directory and avoid deleting it.

Change-Id: Ia595c8bf3f423c1ad5d6faa183a29598c07a11f9
BUG: 1402369
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: http://review.gluster.org/16052
Reviewed-by: Aravinda VK <avishwan@redhat.com>
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
2016-12-07 17:27:02 +05:30
Milind Changire
8e62cff81e tools/glusterfind: xml parsing fix for tiered volumes
gluster volume info <vol> --xml for non-tiered volumes have
'bricks/brick' elements under the 'volInfo/volumes/volume' element.
However, tiered volumes have a 'bricks/hotBricks/brick' and
'bricks/coldBricks/brick' elements under the 'volInfo/volumes/volume'
element.

Fix main.py::get_nodes()

BUG: 1389481
Change-Id: I2f4465bfa8a55e7fa87917d3ec3e69b05d5241b9
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: http://review.gluster.org/15746
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2016-10-27 22:53:43 +05:30
Milind Changire
710cc20bec tools/glusterfind: kill remote processes and separate run-time directories
Problem #1:
Hitting CTRL+C leaves stale processes on remote nodes if glusterfind pre
has been initiated.

Solution #1:
Adding "-t -t" to ssh command-line forces pseudo-terminal to be assigned
to remote process. When local process receives Keyboard Interrupt,
SIGHUP is immediately conveyed to the remote terminal causing remote
changelog.py process to terminate immediately.

Problem #2:
Concurrent glusterfind pre runs are not possible on the same glusterfind
session in case of a runaway process.

Solution #2:
glusterfind pre runs now add random directory name to the working
directory to store and manage temporary database and changelog
processing.
If KeyboardInterrupt is received, the function call
run_cmd_nodes("cleanup", args, tmpfilename=gtmpfilename)
cleans up the remote run specific directory.

Patch:
7571380 cli/xml: Fix wrong XML format in volume get command
broke "gluster volume get <vol> changelog.rollover-time --xml"
Now fixed function utils.py::get_changelog_rollover_time()

Fixed spurious trailing space getting written if second path is empty in
main.py::write_output()
Fixed repetitive changelog processing in changelog.py::get_changes()

Change-Id: Ia8d96e2cd47bf2a64416bece312e67631a1dbf29
BUG: 1382236
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: http://review.gluster.org/15609
Smoke: Gluster Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2016-10-17 12:16:36 +05:30
Kaleb S. KEITHLEY
2f81ada6f1 core: use readdir(3) with glibc, and associated cleanup
Starting with glibc-2.23 (i.e. what's in Fedora 25), readdir_r(3)
is marked as deprecated. Specifically the function decl in <dirent.h>
has the deprecated attribute, and now warnings are thrown during the
compile on Fedora 25 builds.

The readdir(_r)(3) man page (on Fedora 25 at least) and World+Dog say
that glibc's readdir(3) is, and always has been, MT-SAFE as long as
only one thread is accessing the directory object returned by opendir().
World+Dog also says there is a potential buffer overflow in readdir_r().
World+Dog suggests that it is preferable to simply use readdir(). There's
an implication that eventually readdir_r(3) will be removed from glibc.
POSIX has, apparently deprecated it in the standard, or even removed it
entirely.

Over and above that, our source near the various uses of readdir(_r)(3)
has a few unsafe uses of strcpy()+strcat().

(AFAIK nobody has looked at the readdir(3) implemenation in *BSD to see
if the same is true on those platforms, and we can't be sure of MacOS
even though we know it's based on *BSD.)

Change-Id: I5481f18ba1eebe7ee177895eecc9a80a71b60568
BUG: 1356998
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
Reviewed-on: http://review.gluster.org/14838
Smoke: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
2016-07-07 08:51:08 -04:00
Aravinda VK
15956135bb packaging: Remove ".py" extension from symlink target
S57glusterfind-delete-post.py* is packaged in
$LIBEXEC/glusterfs/glusterfind directory, symlink created to
/var/lib/glusterd/hooks directory

ln -s $(libexecdir)/glusterfs/glusterfind/S57glusterfind-delete-post.py \
$(DESTDIR)$(GLUSTERD_WORKDIR)/hooks/1/delete/post/S57glusterfind-delete-post.py

".py" should be removed from symlink target so that build system will
not try to compile the symlink files and generate .pyc/.pyo files.

BUG: 1356868
Change-Id: I9155378aa72eb8559ba8af76d91a4985f69f38f9
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/14928
Smoke: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
2016-07-15 13:15:21 +05:30
Kaleb S KEITHLEY
4c56d69dd8 scripts: bash-isms in scripts
Change-Id: I05f8718fb73c16dbb37c1f7460bb7b55cc0ffe97
BUG: 1336853
Reported-by: Patrick Matthäi <pmatthaei@debian.org>
Signed-off-by: Kaleb S KEITHLEY <kkeithle@redhat.com>
Reviewed-on: http://review.gluster.org/14401
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
Smoke: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
2016-05-17 11:06:07 -04:00
Milind Changire
bd0062713a tools/glusterfind: add --full option to query command
The --full option will use brickfind.py to list all files in the volume.
The output file will contain url-encoded file names prefixed with the
tag string to indicate that all files should be considered as new.
The default tag string for the --full option is "NEW". This can be
changed with the --tag-for-full-find command-line option.

Change-Id: Ic85ba5db062e19df13ae9dc2de8a08eacb5c9792
BUG: 1286279
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: http://review.gluster.org/12779
Smoke: Gluster Build System <jenkins@build.gluster.com>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2015-11-28 14:56:02 +05:30