1
0
mirror of https://github.com/gluster/glusterfs.git synced 2026-02-05 15:48:40 +01:00

100 Commits

Author SHA1 Message Date
Pranith Kumar K
9ecbd69127 cluster/afr: Heal directory rename without rmdir/mkdir
Problem1:
When a directory is renamed while a brick
is down entry-heal always did an rm -rf on that directory on
the sink on old location and did mkdir and created the directory
hierarchy again in the new location. This is inefficient.

Problem2:
Renamedir heal order may lead to a scenario where directory in
the new location could be created before deleting it from old
location leading to 2 directories with same gfid in posix.

Fix:
As part of heal, if oldlocation is healed first and is not present in
source-brick always rename it into a hidden directory inside the
sink-brick so that when heal is triggered in new-location shd can
rename it from this hidden directory to the new-location.

If new-location heal is triggered first and it detects that the
directory already exists in the brick, then it should skip healing the
directory until it appears in the hidden directory.

Credits: Ravi for rename-data-loss.t script

Fixes: #1211
Change-Id: I0cba2006f35cd03d314d18211ce0bd530e254843
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
2020-10-01 12:03:25 +00:00
Amar Tumballi
097db13c11 tests: provide an option to mark tests as 'flaky'
* also add some time gap in other tests to see if we get things properly
* create a directory 'tests/000/', which can host any tests, which are flaky.
* move all the tests mentioned in the issue to above directory.
* as the above dir gets tested first, all flaky tests would be reported quickly.
* change `run-tests.sh` to continue tests even if flaky tests fail.

Reference: gluster/project-infrastructure#72
Updates: #1000
Change-Id: Ifdafa38d083ebd80f7ae3cbbc9aa3b68b6d21d0e
Signed-off-by: Amar Tumballi <amar@kadalu.io>
2020-08-20 08:01:07 +00:00
Mohit Agrawal
3204091c85 test: ./tests/features/ssl-ciphers.t fail on centos 8
Check the tlsv1 openssl connection based on openssl version.
If openssl version is 1.1 it supports tls1 protocol otherwise
it supports tlsv1_2 protocol.

Fixes: #1403
Change-Id: I3ca286492049e6f84de70e3b969fa41db10378ab
Signed-off-by: Mohit Agrawal <moagrawa@redhat.com>
2020-07-31 04:39:07 +00:00
Csaba Henk
9ec6407285 tests/features/interrupt.t: fixes
- Modify the patterns for which we grep the logs so
  that they don't match themselves. The test runner
  inserts the invocation of the cases to the log, thus
  the patterns will occur in the logs verbatim. So if
  the pattern matches itself, the test case will be
  moot (always reporting success).

- Invoke the test utility (open-and-sleep) on
  unique paths so that the file at the passed
  path shall be created on each invocation.

  The kernel does not send an interrupt if
  the file is extant. (This was shadowed by
  the above mistske with result evaluation.)

- Modify the pattern for which we grep the log in
  the test case where interrupt handling is expected
  so that it asserts that the interrupt was handled.
  (So far we did not exclude the possibility of the
  interrupt triggered but not handled due to a race;
  however, it seems to be the case that this theoretic
  race does not have the potential to prevent interrupt
  handling. And if this ever changes in the future we'd
  rather be notified about that.)

Change-Id: I606da2b4064c1ecc4781c7dfdefed95a433478ce
Updates: #1374
Signed-off-by: Csaba Henk <csaba@redhat.com>
2020-07-19 04:32:10 +00:00
Pranith Kumar K
3510916573 mount/fuse: use cookies to get fuse-interrupt-record instead of xdata
Problem:
On executing tests/features/flock_interrupt.t the following error log
appears
[2020-06-16 11:51:54.631072 +0000] E
[fuse-bridge.c:4791:fuse_setlk_interrupt_handler_cbk] 0-glusterfs-fuse:
interrupt record not found

This happens because fuse-interrupt-record is never sent on the wire by
getxattr fop and there is no guarantee that in the cbk it will be
available in case of failures.

Fix:
wind getxattr fop with fuse-interrupt-record as cookie and recover it
in the cbk

Fixes: #1310
Change-Id: I4cfff154321a449114fc26e9440db0f08e5c7daa
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
2020-06-18 06:12:07 +00:00
Mohit Agrawal
39f4a03cba tests: Fix for spurious failure for some test cases
Problem: Sometimes test case is failing at the time of creating files
         on mount point after mounting the volume

Solution: After started the volume need to wait to make sure all
          bricks instances are completely started so put a online_brick_count
          check after just started the volume

Change-Id: I5020e7e417539377277ca00189f9c51d2cf877a6
Fixes: #1162
Signed-off-by: Mohit Agrawal <moagrawal@redhat.com>
2020-04-16 11:15:51 +00:00
Rinku Kothiya
43ed6fa34b tests: Fix spurious failure of worm.t
When the output of date command is a single digit
number it is preceded by zero which is getting
considered as an octal number. Removing the leading
zero from the number solved the problem.

Fixes: #1156

Change-Id: Iac4fa20607c0bb90d94dd8ff157ef6b60932c560
Signed-off-by: Rinku Kothiya <rkothiya@redhat.com>
2020-04-13 14:51:55 +00:00
Mohit Agrawal
fb20713b38 Posix: Use simple approach to close fd
Problem: posix_release(dir) functions add the fd's into a ctx->janitor_fds
         and janitor thread closes the fd's.In brick_mux environment it is
         difficult to handle race condition in janitor threads because brick
         spawns a single janitor thread for all bricks.

Solution: Use synctask to execute posix_release(dir) functions instead of 
          using background a thread to close fds.

Credits: Pranith Karampuri <pkarampu@redhat.com>
Change-Id: Iffb031f0695a7da83d5a2f6bac8863dad225317e
Fixes: bz#1811631
Signed-off-by: Mohit Agrawal <moagrawal@redhat.com>
2020-03-20 04:08:42 +00:00
David Spisla
476139c962 WORM-Xlator: Initial write of a file succeeds if auto-commit-period 0
If worm-file-level enabled and auto-commit-period 0 an initial write
of a file (e.g. $ echo test >> file1.txt) would lead to an zero byte
file because the WORM xlator immediately WORMed the file when it was
created.

To avoid this we move the setting of trusted.worm_file from
worm_create_cbk to worm_release . This means that this xattr will set
when the filehandle is closed and all initial WRITE FOPs succeed.

Finally we also perform gf_worm_state_transition in worm_release to
ensure that the file will be immediately WORMed after the file handle
was closed.

Change-Id: I5d02e18975b646ca1a27ed41d836e9d0dc333204
Fixes: bz#1808421
Signed-off-by: David Spisla <david.spisla@iternity.com>
2020-03-17 04:24:14 +00:00
Mohit Agrawal
1e7fc7e540 ssl/test: ssl test case is failing when using specific cipher
Problem: On RHEL-8 ssl test case is failing when trying to
         connect with a peer after using the specific cipher.

Solution: If cipher is not supported by openssl on rhel-8 then
          test case is failed.To avoid the issue validate the
          cipher before connecting with peer.

Change-Id: I96d92d3602cf7fd40337126c8305a3f8925faf9b
Fixes: bz#1756900
Signed-off-by: Mohit Agrawal <moagrawal@redhat.com>
2019-11-15 07:38:05 +00:00
l17zhou
54ed71dba1 rpc: Cleanup SSL specific data at the time of freeing rpc object
Problem: At the time of cleanup rpc object ssl specific data
         is not freeing so it has become a leak.

Solution: To avoid the leak cleanup ssl specific data at the
          time of cleanup rpc object

Credits: l17zhou <cynthia.zhou@nokia-sbell.com.cn>
Fixes: bz#1768407
Change-Id: I37f598673ae2d7a33c75f39eb8843ccc6dffaaf0
2019-11-08 01:31:18 +00:00
Mohammed Rafi KC
f221ba83e1 ssl/test: Change the rsa key length to 2048
On a rhel-8 machine, we need to have a key length
greater than or eaual to 2048. So changing the values
to 2048 to pass the test.

Credits: Mohit Agrawal <moagrawal@redhat.com>

Change-Id: I0f21db4d737203d0b2e44e7e61f50ae1279795ad
Updates: bz#1756900
Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com>
2019-10-29 14:57:02 +00:00
Vishal Pandey
47465050c4 Worm: xattr update on changing access time of a WORM-Retained file
Retention-period must be updated on changing the access time
of a worm-retained file. Retention-period must be changed in the
"trusted.reten-state" xattr

Change-Id: Ieab758a4cf6da3b4bb1d6a3e4f95f400c8a11f1d
Fixes: bz#1554286
2019-10-23 04:04:22 +00:00
Mohit Agrawal
25ad5aca23 tests: subdir-mount.t is failing for brick_mux regrssion
To avoid the failure wait to run hook script S13create-subdir-mounts.sh
after executed add-brick command by test case.

Change-Id: I063b6d0f86a550ed0a0527255e4dfbe8f0a8c02e
fixes: bz#1720993
Signed-off-by: Mohit Agrawal <moagrawal@redhat.com>
2019-06-17 16:13:08 +05:30
Amar Tumballi
c525fdc6bd tests: keep glfsxmp in tests directory
this is critical so all the tests will be contained in the same
directory, and one can just 'cp -a tests/ <any-location>/' and
run glusterfs tests.

only 'glfsxmp.c' was an exception as it was just copying the
file from api example directory. Now moved it to tests.

updates: bz#1193929
Change-Id: I00359d64be580bffc5b3c3a090968d86c2c6952a
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2019-06-11 13:26:59 +00:00
Amar Tumballi
abc4d4cfa0 lcov: run more fops on translators
Translators covered:
* playground/template
* debug/delay-gen
* debug/error-gen
* features/namespace
* features/quiesce
* meta

updates: bz#1693692
Change-Id: Ic8fde8efcb309ea492d8e819241f786f7ff467a1
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2019-06-04 09:52:59 +00:00
Milind Changire
06fa261207 socket/ssl: fix crl handling
Problem:
Just setting the path to the CRL directory in socket_init() wasn't working.

Solution:
Need to use special API to retrieve and set X509_VERIFY_PARAM and set
the CRL checking flags explicitly.
Also, setting the CRL checking flags is a big pain, since the connection
is declared as failed if any CRL isn't found in the designated file or
directory. A comment has been added to the code appropriately.

Change-Id: I8a8ed2ddaf4b5eb974387d2f7b1a85c1ca39fe79
fixes: bz#1687326
Signed-off-by: Milind Changire <mchangir@redhat.com>
2019-03-19 09:38:28 +00:00
Csaba Henk
35ac5ae13f fuse lock interrupt: fix flock_interrupt.t
updates: bz#1193929
Change-Id: I347de62755100cd69e3cf341434767ae23fd1ba4
Signed-off-by: Csaba Henk <csaba@redhat.com>
2019-03-05 02:49:08 +00:00
Csaba Henk
eb77d69be5 fuse: SETLKW interrupt
Use the (f)getxattr based clearlocks interface to
interrupt a pending lock request.

updates: #465
Change-Id: I4e91a4d8791fc688fed400a02de4c53487e61be2
Signed-off-by: Csaba Henk <csaba@redhat.com>
2018-12-14 18:11:01 +00:00
Amar Tumballi
d49b41e817 fuse: add --lru-limit option
The inode LRU mechanism is moot in fuse xlator (ie. there is no
limit for the LRU list), as fuse inodes are referenced from
kernel context, and thus they can only be dropped on request of
the kernel. This might results in a high number of passive
inodes which are useless for the glusterfs client, causing a
significant memory overhead.

This change tries to remedy this by extending the LRU semantics
and allowing to set a finite limit on the fuse inode LRU.

A brief history of problem:

When gluster's inode table was designed, fuse didn't have any
'invalidate' method, which means, userspace application could
never ask kernel to send a 'forget()' fop, instead had to wait
for kernel to send it based on kernel's parameters. Inode table
remembers the number of times kernel has cached the inode based
on the 'nlookup' parameter. And 'nlookup' field is not used by
no other entry points (like server-protocol, gfapi etc).

Hence the inode_table of fuse module always has to have lru-limit
as '0', which means no limit. GlusterFS always had to keep all
inodes in memory as kernel would have had a reference to it.
Again, the reason for this is, kernel's glusterfs inode reference
was pointer of 'inode_t' structure in glusterfs. As it is a
pointer, we could never free it (to prevent segfault, or memory
corruption).

Solution:

In the inode table, handle the prune case of inodes with 'nlookup'
differently, and call a 'invalidator' method, which in this case is
fuse_invalidate(), and it sends the request to kernel for getting
the forget request.

When the kernel sends the forget, it means, it has dropped all
the reference to the inode, and it will send the forget with the
'nlookup' parameter too. We just need to make sure to reduce the
'nlookup' value we have when we get forget. That automatically
cause the relevant prune to happen.

Credits: Csaba Henk, Xavier Hernandez, Raghavendra Gowdappa, Nithya B

fixes: bz#1560969
Change-Id: Ifee0737b23b12b1426c224ec5b8f591f487d83a2
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2018-12-14 17:34:28 +00:00
Soumya Koduri
ad35193718 gfapi: Offload callback notifications to synctask
Upcall notifications are received from server via epoll
and same thread is used to forward these notifications
to the application. This may lead to deadlock and hang
in the following scenario.

Consider if as part of handling these callbacks,
application has to do some operations which involve
sending I/Os to gfapi stack which inturn have to wait for
epoll threads to receive repsonse. Thus this may lead to
deadlock if all the epoll threads are waiting to complete
these callback notifications.

To address it, instead of using epoll thread itself,
make use of synctask to send those notificaitons to the
application.

Change-Id: If614e0d09246e4279b9d1f40d883a32a39c8fd90
updates: bz#1648768
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
2018-11-26 04:20:05 +00:00
Soumya Koduri
c2e758b54d lease: Treat unlk request as noop if lease not found
When the glusterfs server recalls the lease, it expects
client to flush data and unlock the lease. If not it sets
a timer (starting from the time it sends RECALL request) and post
timeout, it revokes it.

Here we could have a race where in client did send UNLK
lease request but because of network delay it may have reached
after server revokes it. To handle such situations, treat
such requests as noop and return sucesss.

Change-Id: I166402d10273f4f115ff04030ecbc14676a01663
updates: bz#1648768
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
2018-11-17 09:16:13 +00:00
Amar Tumballi
c3fcff9ccb glupy: remove from the build
Based on the proposal to remove few features as they are not
actively maintained [1], removing 'glupy' translator from the
build.

[1] https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html

This patch aims at clearing the translator from build and tests.
A followup is needed to remove the code from repository.

Updates: bz#1642810
Change-Id: I41d0c1956330c3bbca62c540ccf9ab01bbf3a092
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2018-11-08 02:46:48 +00:00
Amar Tumballi
bab025b08e tests/interrupt.t: remove 'stripe' volume type
Merged the patch which introduced this testcase after the
'remove stripe' patch got merged, and hence the confusion.

Updates: bz#1193929
Change-Id: Ia08552debb111292caf14e51ea6a27334fe5c788
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2018-11-06 16:25:27 +05:30
Csaba Henk
4c6b063463 fuse: diagnostic FLUSH interrupt
We add dummy interrupt handling for the FLUSH
fuse message. It can be enabled by the
"--fuse-flush-handle-interrupt" hidden command line
option, or "-ofuse-flush-handle-interrupt=yes"
mount option.

It serves no other than diagnostic & demonstational
purposes -- to exercise the interrupt handling framework
a bit and to give an usage example.

Documentation is also provided that showcases interrupt
handling via FLUSH.

Change-Id: I522f1e798501d06b74ac3592a5f73c1ab0590c60
updates: #465
Signed-off-by: Csaba Henk <csaba@redhat.com>
2018-11-06 04:21:57 +00:00
Kaleb S. KEITHLEY
5b91555a58 python3: assume python3 unless building _packages_ on sys without py3
The jenkins release-new job runs on a CentOS 7 box, which does not
have python3. As a result it runs (autogen.sh and) configure before
producing the dist tar file, converting all the python3 shebangs to
python2 shebangs in the dist tar file.

Then when that tar file is "carried" to, e.g. Fedora koji build
system to build packages, the shebangs are incorrect, despite having
originally been correct in the git repo.

Change-Id: I5154baba3f6d29d3c4823bafc2b57abecbf90e5b
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-09-27 03:12:58 +00:00
Amar Tumballi
d4770602e0 dht: utilize the framework to pass-through xlator tasks
Also fixes the issue caused due to not converting back the fn function
to after getting its address. We wanted the value of the field, not
the address of the pt_fop field.

With this patch, DHT will always be started in pass-through
mode if the number of subvols is just 1.

Fixes some tests to make sure DHT is in full config (ie, subvols > 1).
  - increased timeout of brick-mux test as it was bordering on 300 seconds.
  - Also change the volume type to supported 'replica 3' from 'replica 2'.
  - also no DHT tests should assume presence of DHT when there is
    just 1 brick in volume

Credits: Nithya B <nbalacha@redhat.com>

fixes: #405
Change-Id: I8e55239ce58d6ac6ae1901e2e384be1ecbd33d6e
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2018-09-19 04:26:55 +00:00
Gluster Ant
e16868dede Land part 2 of clang-format changes
Change-Id: Ia84cc24c8924e6d22d02ac15f611c10e26db99b4
Signed-off-by: Nigel Babu <nigelb@redhat.com>
2018-09-12 17:52:45 +05:30
Kaleb S. KEITHLEY
af0d5a9b53 core: python3
see https://review.gluster.org/#/c/19788/,
    https://review.gluster.org/#/c/19871/,
    https://review.gluster.org/#/c/19952/,
    https://review.gluster.org/#/c/20104/,
    https://review.gluster.org/#/c/20162/,
    https://review.gluster.org/#/c/20185/,
    https://review.gluster.org/#/c/20207/,
    https://review.gluster.org/#/c/20227/,
    https://review.gluster.org/#/c/20307/,
    https://review.gluster.org/#/c/20320/,
    https://review.gluster.org/#/c/20332/,
    https://review.gluster.org/#/c/20364/,
    https://review.gluster.org/#/c/20441/, and
    https://review.gluster.org/#/c/20484

shebangs changed from /usr/bin/python2 to /usr/bin/python3.
(Reminder, various distribution packaging guidelines require use
of explicit python version and don't allow '#!/usr/bin/env python',
regardless of how handy that idiom may be.)

glusterfs.spec(.in) package python{2,3}-gluster and python2 or
python3 dependencies as appropriate.

configure(.ac):
+ test for and use python2 or python3 as appropriate. If build
  machine has python2 and python3, use python3. Override by
  setting PYTHON=/usr/bin/python2 when running configure.
+ PYTHONDEV_CPPFLAGS from python[23]-config --includes is a
  better match to the original python sysconfig.get_python_inc().
  All those other extraneous flags breaks the build.
+ Only change the shebangs once. Changing them over and over
  again, e.g., during a `make glusterrpms` in extras/LinuxRPM
  just sends make (is it really make that's looping?) into an
  infinite loop. If you figure out why, let me know.
+ Oldest python2 is python2.6 on CentOS 6 and Debian 8 (Jessie).
  Everything else has 2.7 or 3.x
+ logic from https://review.gluster.org/c/glusterfs/+/21050, which
  needs to be removed/merged after that patch is merged.

Builds on CentOS 6, CentOS 7, Fedora 28, Fedora rawhide, and the
mysterious RHEL > 7.

Change-Id: Idae21d3b6f58b32372e1daa0d234e491e563198f
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-09-03 09:14:44 +00:00
Yaniv Kaul
621138ce76 All: run codespell on the code and fix issues.
Please review, it's not always just the comments that were fixed.
I've had to revert of course all calls to creat() that were changed
to create() ...

Only compile-tested!

Change-Id: I7d02e82d9766e272a7fd9cc68e51901d69e5aab5
updates: bz#1193929
Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
2018-07-22 14:40:16 +00:00
Kaleb S. KEITHLEY
fe9b724856 core/various: python3 compat, prepare for python2 -> python3
see https://review.gluster.org/#/c/19788/,
    https://review.gluster.org/#/c/19871/,
    https://review.gluster.org/#/c/19952/,
    https://review.gluster.org/#/c/20104/,
    https://review.gluster.org/#/c/20162/,
    https://review.gluster.org/#/c/20185/,
    https://review.gluster.org/#/c/20207/,
    https://review.gluster.org/#/c/20227/, and
    https://review.gluster.org/#/c/20307/

This patch fixes more selected comma white space (ws_comma) as suggested
by the 2to3 utility.

Note: Fedora packaging guidelines and SUSE rpmlint require explicit
shebangs, so popular practices like #!/usr/bin/env python and
or #!/usr/bin/python3

Note: Selected small fixes from 2to3 utility. Specifically apply,
basestring, funcattrs, has_key, idioms, map, numliterals, raise,
set_literal, types, urllib, and zip have already been applied. Also
version agnostic imports for urllib, cpickle, socketserver, _thread,
queue, etc., suggested by Aravinda in https://review.gluster.org/#/c/19767/1

Note: these 2to3 fixes report no changes are necessary: asserts, buffer,
exec, execfile, exitfunc, filter, getcwdu, imports2, input, intern,
itertools, metaclass, methodattrs, ne, next, nonzero, operator, paren,
raw_input, reduce, reload, renames, repr, standarderror, sys_exc, throw,
tuple_params, xreadlines.

Change-Id: I60932030813484803f73733a9b2b7b23c7a843fd
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-06-21 05:45:08 +00:00
Kaleb S. KEITHLEY
202d27c530 core/various: python3 compat, prepare for python2 -> python3
see https://review.gluster.org/#/c/19788/ and
    https://review.gluster.org/#/c/19871/

Selected small fixes from 2to3 utility. Specifically apply, basestring,
funcattrs, idioms, numliterals, set_literal, types, urllib, zip

Note: these 2to3 fixes report no changes are necessary: exec, execfile,
exitfunc, filter, getcwdu, intern, itertools, metaclass, methodattrs, ne,
next, nonzero, operator, paren, raw_input, reduce, reload, renames, repr,
standarderror, sys_exc, throw, tuple_params, xreadlines.

Any 2to3 fixes not in the above two lists have more extensive changes
which will follow in separate patches.

most unicode changes suggested by 2to3 will need to be applied at the
same time as changing the shebangs from python2 to python3. Prashanth
notes that unicode strings in py2 need 'u' prefix; unicode strings in
py3 3.0, 3.1, and 3.2 a 'u' prefix will throw an error, but in py3 3.3+
it is legal (or just ignored). All Linux dists we care about have 3.3
or later so we can leave 'u' prefixes on unicode strings.

Change-Id: I49bba2f328b0ee24b9a8115a7183be979981563e
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-05-30 03:37:46 +00:00
Poornima G
80262e2984 tests: Add lease test case
Updates: #350
Change-Id: Iee78ab4baf48c481de1e13ff2b0393bc106b7d0e
Signed-off-by: Poornima G <pgurusid@redhat.com>
Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
2018-05-05 11:53:59 +00:00
Kaleb S. KEITHLEY
4e7b423d3c core/various: python3 compat, prepare for python2 -> python3
see https://review.gluster.org/#/c/19788/

use print fn from __future__

Change-Id: If5075d8d9ca9641058fbc71df8a52aa35804cda4
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-05-02 11:28:46 +00:00
Kaleb S. KEITHLEY
29024cfdd5 core/build/various: python3 compat, prepare for python2 -> python3
Note 1) we're not supposed to be using #!/usr/bin/env python, see
https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#Shebang_lines

Note 2) we're also not supposed to be using "!/usr/bin/python,
see https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build#Quick_Opt-Out

The previous patch (https://review.gluster.org/19767) tried to do too
much in one patch, so it was abandoned.

This patch does two things:
1) minor cleanup of configure(.ac) to explicitly use python2
2) change all the shebang lines to #!/usr/bin/python2 and add them
where they were missing based on warnings emitted during rpmbuild.

In a follow-up patch python2 will eventually be changed to python3.

Before that python2-isms (e.g. print, string.join(), etc.) need to be
converted to python3. Some of those can be rewritten in version agnostic
python. E.g. print statements become print() with "from __future_ import
print_function". The python 2to3 utility will be used for some of those.
Also Aravinda has given guidance in the comments to the first patch for
changes.

updates: #411
Change-Id: I471730962b2526022115a1fc33629fb078b74338
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-04-12 11:04:27 +00:00
N Balachandran
09ff538b68 cluster/dht: enable lookup-optimize by default
Lookup-optimize has been shown to improve create
performance. The code has been in the project for several
years and is considered stable.

Enabling this by default in order to test this in the
upstream regression runs.

Change-Id: Iab792979ee34f0af4713931e0b5b399c23f65313
updates: bz#1557435
BUG: 1557435
Signed-off-by: N Balachandran <nbalacha@redhat.com>
2018-04-04 03:03:34 +00:00
Niklas Hambüchen
0056feaa21 python: Remove all uses of find_library. Fixes #1450593
`find_library()` doesn't consider LD_LIBRARY_PATH on Python < 3.6.

Change-Id: Iee26085cb5d14061001f19f032c2664d69a378a8
BUG: 1450593
Signed-off-by: Niklas Hambüchen <mail@nh2.me>
2018-03-24 05:10:31 +00:00
Amar Tumballi
685d4409f9 hooks: add a script to stat the subdirs in add-brick
The subdirectories are expected to be present for a subdir
mount to be successful. If not, the client_handshake()
itself fails to succeed. When a volume is about to get
mounted first time, this is easier to handle, as if the
directory is not present in one brick, then its mostly
not present in any other brick. In case of add-brick,
the directory is not present in new brick, and there is
no chance of healing it from the subdirectory mount, as
in those clients, the subdir itself will be 'root' ('/')
of the filesystem. Hence we need a volume mount to heal
the directory before connections can succeed.

This patch does take care of that by healing the directories
which are expected to be mounted as subdirectories from the
volume level mount point.

Change-Id: I2c2ac7b7567fe209aaa720006d09b68584d0dd14
BUG: 1549915
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2018-03-06 14:41:44 +00:00
Vishal Pandey
dbd94d5bee features/worm: new config option to manage deletion of Worm files.
Add a new configuration option worm-files-deletable to
 file-level Worm in order to control behaviour of Worm files upon deletion.

Steps to Test:
1. Add all the configuration options to a volume to activate file-level-worm
2. Option features.worm-files-deletable is set to 1 by default.
3. Create a new file and wait for the retention time to expire.
4. After retention time expires, do an truncate, rename, unlink, link
   or write to send the file in Worm state.
5. After that do `rm -f filename`.
6. The file is successfully removed.
7. Repeat from step 2 by setting features.worm-files-deletable 0.
   This time deletion should not be successful.

Change-Id: Ibc89861ee296e065330b93a9f9606be5da40af31
BUG: 1508898
Signed-off-by: Vishal Pandey <vishpandey2014@gmail.com>
2017-11-20 11:41:27 +00:00
Vishal Pandey
4edea12891 write once read many: file appendable in worm state
Issue:
A new file is appendable even when file level worm is 1.

Fix:
Do a state transition in writev function in worm.c file.

Steps To Test:
1- Activate file level worm.
2- Create a new file.
3- Leave file dormant for auto commit period.
4- Try and append some content to the file.
5- check the file if new content has been appended or not.
6- check if file has been transitioned to Worm Retention state.

Change-Id: I52d50ad888cb0c39ad54be9352ccb07d48b8d71a
BUG: 1505807
Signed-off-by: Vishal Pandey <vishpandey2014@gmail.com>
2017-10-30 16:00:46 +00:00
Amar Tumballi
9aa574a51b protocol/client: handle the subdir handshake properly for add-brick
There should be different way we handle handshake in case of subdir
mount for the first time, and in case of subsequent graph changes.

Change-Id: I2a7ba836433bb0a0f4a861809e2bb0d7fbc4da54
BUG: 1505323
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2017-10-29 08:55:42 +01:00
Nigel Babu
6cad2cd12b Disable failing NetBSD tests
BUG: 1501390
Change-Id: I9a04c094783ec33e617baeae3d0e0cbedb1d6c3b
2017-10-13 16:24:43 +00:00
Amar Tumballi
a3767c4ceb protocol-auth: use the proper validation method
Currently, server protocol's init and glusterd's option
validation methods are different, causing an issue. They
should be same for having consistent behavior

Updates #175

Change-Id: Ibbf9a18c7192b2d77f9b7675ae7da9b8d2fe5de4
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2017-10-12 10:36:26 +00:00
luneo7
00a4dc058a features/worm: Adding check to newloc when doing rename
Problem: Since rename didn't check if newloc exists and it's
retention state it was possible to rename a new file that wasn't
in retention over a existing file that was in read-only state.

Change-Id: I63c6bbabb7bb456ebedf201cc77b878ffda62229
BUG: 1484490
Signed-off-by: luneo7 <luneo7@gmail.com>
Reviewed-on: https://review.gluster.org/18104
Tested-by: jiffin tony Thottan <jthottan@redhat.com>
Tested-by: Prashanth Pai <ppai@redhat.com>
Smoke: Gluster Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Karthik U S <ksubrahm@redhat.com>
Reviewed-by: Amar Tumballi <amarts@redhat.com>
2017-09-18 07:14:59 +00:00
Pranith Kumar K
75223c0a1b debug/delay-gen: Implement delay-generation feature
Background:
I was working on a customer issue where the disks were responding some times
after seconds. It was becoming very difficult to recreate the issues in our
labs, so had to come up with this feature.

Requirements:
We need an xlator which can delay x% of ops for y micro seconds.
We should be able to enable delays for specific fops.

This feature is modeled after error-gen. Most of the logic
is borrowed from that xlator. This is a minimum implementation
of the feature which satisfied the requirements I had. May be
in future with more requirements and understanding of the problem
further we can improve upon this implementation.

Here are the commands and what they do:
Enable delay-gen: (This is similar to how err-gen is enabled on the brick side)
- gluster volume set <volname> delay-gen posix

Set the percentage of fops that need to be delayed
- gluster volume set <volname> delay-gen.delay-percentage 50
Default is 10%

Set the delay in micro seconds
- gluster volume set <volname> delay-gen.delay-duration 500000
Default is 100000

Set comma separated fops to be delayed
- gluster v set r2 delay-gen.enable read,write
Default is all fops.

Fixes #257
Change-Id: Ib547bd39cc024c9cdb63754d21e3aa62fc9d6473
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: https://review.gluster.org/17591
Smoke: Gluster Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Jeff Darcy <jeff@pl.atyp.us>
2017-08-31 17:17:09 +00:00
N Balachandran
96b33b4b27 cluster/dht: Reorder dir operations in gf_defrag_fix_layout
Earlier, rebalance performed a fix-layout on a directory
before healing its subdirectories. If there were a lot of
subdirs, it could take a while before all subdirs were
created on the newly added bricks. As dht_readdirp only lists
dirs from their hashed subvol, those dirs which hashed to
the newly added bricks but were not yet created on them were
not listed.

Now, the child dirs are listed and processed before the layout
of the parent is fixed. This introduces a change in behaviour
where files in subdirs are migrated before those in parent
directories.

Credit: Shyam <srangana@redhat.com>
Github issue: #239

Change-Id: I8ae7f24a510754cd8d1b31e5d608bcf1928599e2
BUG: 1248393
Signed-off-by: N Balachandran <nbalacha@redhat.com>
Reviewed-on: https://review.gluster.org/18045
Smoke: Gluster Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
2017-08-20 02:47:02 +00:00
Amar Tumballi
590ae48c65 glusterfsd: allow subdir mount
Changes:

1. Take subdir mount option in client (mount.gluster / glusterfsd)
2. Pass the subdir mount to server-handshake (from client-handshake)
3. Handle subdir-mount dir's lookup in server-first-lookup and handle
   all fops resolution accordingly with proper gfid of subdir
4. Change the auth/addr module to handle the multiple subdir entries
   in option, and valid parsing.

How to use the feature:

`# mount -t glusterfs $hostname:/$volname/$subdir /$mount_point`
Or
`# mount -t glusterfs $hostname:/$volname -osubdir_mount=$subdir /$mount_point`

Option can be set like:

`# gluster volume set <volname> auth.allow "/subdir1(192.168.1.*),/(192.168.10.*),/subdir2(192.168.8.*)"`

Updates #175

Change-Id: I7ea57f76ddbe6c3862cfe02e13f89e8a39719e11
Signed-off-by: Amar Tumballi <amarts@redhat.com>
Reviewed-on: https://review.gluster.org/17141
Smoke: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
2017-08-04 05:26:42 +00:00
Niels de Vos
d26f0bac14 tests: disable IPC test-case
The IPC test-case is not correct, and can cause segfaults or hangs. The
ipctest.py script calls glfs_ipc() with the `glfs_t` structure, but
should do so with a `glfs_fd_t`.

In addition, the test-case is written in a way that we do not suggest to
use libgfapi. Python scripts are encouraged to use the bindings from the
libgfapi-python project. It would be better to rewrite the test in C so
that there is type-checking while compiling and no additional issues
with portability (see `LD_PRELOAD` note in the `.t` file).

Change-Id: Icb52b5b1585fbee98f2c694547c31df0aa2ba70b
Updates: #269
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: https://review.gluster.org/17786
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Jeff Darcy <jeff@pl.atyp.us>
Reviewed-by: Raghavendra Talur <rtalur@redhat.com>
2017-07-18 11:13:34 +00:00
Raghavendra G
d5865881de tests/lock_revocation: mark as bad
The test is failing in master. see gluster-devel for more details.

Change-Id: I7a589ad2c54bd55d62f4e66fdf8037c19fc123ea
BUG: 1448364
Signed-off-by: Raghavendra G <rgowdapp@redhat.com>
Reviewed-on: https://review.gluster.org/17234
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Jeff Darcy <jeff@pl.atyp.us>
2017-05-10 14:56:54 +00:00
Amar Tumballi
c1e61fd5df worm: add check for internal processes in ftruncate()
The patch fixes the recently seen issues with worm_sh.t test.

RCA:

-
$ git log --oneline xlators/features/read-only/src/worm.c
1b01bdc worm: allow Self-heal-Daemon to perform some operations
c5a4a77 features/worm: Adding implementation for ftruncate
-

These two patches were merged in reverse order of their submission,
and hence the check added for internal processes got missed in
new fop 'ftruncate()'. The worm_sh.t passed the tests as while
that patch got submitted there was no ftruncate() in worm xlator.

Change-Id: I81a8a45fa2679917a2c859c4f5224a2c3edbc784
BUG: 1423413
Signed-off-by: Amar Tumballi <amarts@redhat.com>
Reviewed-on: https://review.gluster.org/17048
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Zhou Zhengping <johnzzpcrystal@gmail.com>
Reviewed-by: David Spisla <david.spisla@iternity.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
2017-04-18 14:35:12 -04:00