Introduction

While working on an Oracle Exadata Cell Server training environment, I encountered the following CellCLI error:

CELL-01519: Cannot talk to the Management Server (MS). Error: ;
nested exception is:
HTTP transport error:
javax.xml.soap.SOAPException:
java.security.PrivilegedActionException:
javax.xml.soap.SOAPException:
Bad response: 404 Not Found

At first glance, this looked like a Management Server connectivity or HTTP configuration problem.

However, further investigation revealed that the real root cause was:

/opt filesystem = 100% full

The Management Server was running as a process, but it could not create required files because the filesystem had no free space.

This article walks through the complete troubleshooting process.

1. Environment

The issue occurred on a simulated/training Exadata Cell Server environment running:

CellCLI Release: 11.2.3.2.1
Operating System: Oracle Linux
Filesystem: ext3

The Cell Server contained simulated raw storage under:

/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/

The CellCLI components involved were:

RS       Restart Server
MS       Management Server
CELLSRV  Cell Server

2. Initial Symptom

When executing:

LIST CELLDISK;

CellCLI returned:

CELL-01519: Cannot talk to the Management Server (MS).

Error:
HTTP transport error:
javax.xml.soap.SOAPException:
java.security.PrivilegedActionException:
javax.xml.soap.SOAPException:
Bad response: 404 Not Found

The same problem occurred with:

LIST PHYSICALDISK;

and other CellCLI commands.

3. First assumption: Management Server is down

The first thing to check is whether the Cell Server services are running.

Use:

service celld status

Depending on the environment, you may also use:

/etc/init.d/celld status

The result showed:

rsStatus:       running
msStatus:       running
cellsrvStatus:  running

This was important.

It meant:

RS       → Running
MS       → Running
CELLSRV  → Running

So simply saying "MS is down" would have been incorrect.

4. Check the MS process

The Management Server in this older environment runs through Java/OC4J.

Check the process:

ps -ef | grep '[m]s'

We found the Java process associated with:

oc4j.jar

This confirmed that an MS process existed.

However:

A running process does not necessarily mean that the application is healthy.

This became important later.

5. Check listening ports

Next, inspect the listening ports:

ss -lntp

If ss is unavailable:

netstat -lntp

The Java MS process was listening on local ports including:

127.0.0.1:23943
127.0.0.1:23791
127.0.0.1:8888

CELLSRV was also listening on its storage service port.

This indicated that the Java process itself was alive.

6. Test the HTTP endpoint

To determine whether the Java HTTP server was responding:

curl -v http://127.0.0.1:8888/

The HTTP server responded successfully.

This gave us another important clue:

Network/HTTP listener
        ↓
      Alive

But:

CellCLI → MS application
        ↓
       404

Therefore, the problem wasn't simply that the Java HTTP listener was unavailable.

7. Check the Management Server logs

This was the turning point.

In this environment, the MS Java process showed its log locations under:

/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log/

Important files included:

ms.lst
ms.err

Check the error log:

tail -100 \
/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log/ms.err

For a larger investigation:

less \
/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log/ms.err

Search specifically for errors:

grep -i "error" \
/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log/ms.err

Also search for filesystem-related messages:

grep -iE "no space|disk|filesystem|write|IOException" \
/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log/ms.err

8. The actual error in the MS log

The MS log showed:

java.io.IOException: No space left on device

It also showed that the application was unable to create deployment-related directories/files.

For example, the application was attempting to create directories under:

.../application-deployments/ms/ms/

but the operation failed with:

No space left on device

This completely changed the direction of the investigation.

The 404 Not Found was a symptom.

The underlying problem was:

Filesystem full

9. Check filesystem utilization

The first Linux command to run when you see:

No space left on device

is:

df -h

In our case:

Filesystem      Size  Used  Avail  Use%  Mounted on
/dev/sda2        43G   43G     0   100% /opt

This immediately confirmed:

/opt = 100% full

10. Check inode utilization too

A common mistake is to assume that every No space left on device error means disk capacity is exhausted.

It can also mean inode exhaustion.

Check:

df -i

Our result was approximately:

/opt
IUse% = 1%

Therefore:

Disk blocks → FULL
Inodes      → NOT FULL

So the issue was definitely storage capacity, not inode exhaustion.

General rule

When you see:

No space left on device

always check both:

df -h
df -i

11. Find what is consuming the filesystem

Next, identify the largest directories.

On older Linux systems, du and sort may not support modern options such as:

sort -h

So use numeric sorting:

du -x -k /opt 2>/dev/null | sort -n | tail -30

Or, if supported:

du -x -k -d 1 /opt 2>/dev/null | sort -n

The investigation showed:

/opt/oracle/.../disks/raw
≈ 43.6 GB

while:

/opt
≈ 44.5 GB

This was the key finding.

12. Inspect the raw storage files

We then investigated:

du -x -k \
/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/* \
2>/dev/null | sort -n

The result showed:

DISK01   ~5 GB
DISK02   ~5 GB
DISK03   ~5 GB
DISK04   ~5 GB
DISK05   ~5 GB
DISK06   ~11.5 GB
DISK07   ~1 GB

FLASH01  ~1 GB
FLASH02  ~1 GB
FLASH03  ~1 GB
FLASH04  ~1 GB

These were not temporary files.

They are the backing files representing the simulated Cell Server storage.

Therefore:

Do not delete these files to recover space.

For example, deleting:

rm DISK06

would effectively destroy simulated Cell Server storage.

13. Check for deleted files still consuming space

Another important Linux troubleshooting technique is:

lsof +L1

This identifies files that have been deleted from the filesystem but are still held open by a process.

In our environment, Java had references to deleted files such as:

MBeanServerEjb.ser
MBeanServerUserEjb.ser

However, their size was:

0 bytes

Therefore, they were not responsible for the 43 GB filesystem exhaustion.

General troubleshooting principle

If:

df -h

says a filesystem is full but:

du

doesn't account for the usage, check:

lsof +L1

because deleted-but-open files can cause exactly that situation.

14. Check the partition layout

We then checked:

fdisk -l /dev/sda

The VM had:

/dev/sda1    /boot
/dev/sda2    /opt
/dev/sda3    /
/dev/sda5    swap

Specifically:

/dev/sda2 → /opt → ~43 GB
/dev/sda3 → /    → ~19 GB

And /opt was:

100% full

while / still had approximately:

15 GB free

15. Check `/etc/fstab`

The mount configuration was:

cat /etc/fstab

It showed:

LABEL=/      /      ext3
LABEL=/opt   /opt   ext3
LABEL=/boot  /boot  ext3

Therefore /opt was a separate filesystem.

This is important:

/       → /dev/sda3 → 15 GB free
/opt    → /dev/sda2 → 0 GB free

Free space on / cannot automatically be used by /opt.

16. Check whether LVM is involved

Run:

which lvm

In our case, lvm existed as a command, but that does not mean /opt is using LVM.

The actual mount was:

/dev/sda2 → /opt

not:

/dev/mapper/...

So this was a traditional partition/filesystem configuration.

17. Root cause

The final root-cause chain was:

Simulated Cell Server
        │
        ▼
Raw disk files stored under /opt
        │
        ▼
/opt filesystem reaches 100%
        │
        ▼
MS tries to create/write files
        │
        ▼
java.io.IOException:
No space left on device
        │
        ▼
MS application cannot initialize/deploy correctly
        │
        ▼
CellCLI cannot communicate with MS
        │
        ▼
CELL-01519
HTTP 404 Not Found

So:

Primary root cause

/opt filesystem was 100% full.

Secondary symptom

CellCLI returned CELL-01519 / HTTP 404 because MS could not operate correctly.

18. Solution

The appropriate solution for this training environment is:

Increase the /opt filesystem capacity.

Because /opt contains the simulated Cell Server raw storage, deleting the disk files is not an acceptable solution.

The VM administrator/lab provider should increase the storage allocated to /opt, or provide additional storage and redesign the filesystem allocation.

In this particular VM, /opt was:

/dev/sda2
43 GB
ext3

and the partition was already immediately followed by /dev/sda3.

Therefore, do not blindly run resize2fs /dev/sda2.

The filesystem cannot be enlarged beyond the partition boundary unless the underlying partition is first enlarged.

19. What NOT to do

When troubleshooting this type of issue, avoid these dangerous shortcuts.

Don't delete raw disk files

rm /opt/oracle/.../disks/raw/DISK01

Those files represent simulated storage.

Don't blindly resize the filesystem

resize2fs /dev/sda2

A filesystem cannot magically grow beyond its partition.

Don't modify partitions blindly

fdisk /dev/sda

Deleting/recreating the wrong partition can destroy /opt or /.

Don't restart services blindly

service celld restart

A restart won't fix a filesystem that is still 100% full.

First fix the underlying storage problem.

20. After increasing `/opt`

Once sufficient space has been added:

Verify filesystem

df -h /opt

You should see something like:

Filesystem      Size  Used  Avail  Use%
/dev/sda2        XXG   XXG    XXG   XX%

Also verify inodes:

df -i /opt

Check MS logs again

tail -100 \
/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log/ms.err

Search:

grep -iE "error|exception|no space" \
/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log/ms.err

You want to confirm that new No space left on device errors are no longer occurring.

Check Cell services

service celld status

Expected:

rsStatus:       running
msStatus:       running
cellsrvStatus:  running

Test CellCLI

cellcli

Then:

LIST PHYSICALDISK;

LIST CELLDISK;

LIST GRIDDISK;

If these work, the MS communication problem is resolved.

21. Useful troubleshooting command cheat sheet

Check filesystem capacity

df -h

Check inode usage

df -i

Check filesystem type

df -Th /opt

Check mount configuration

cat /etc/fstab

Check partition layout

fdisk -l /dev/sda

Find large directories

du -x -k /opt 2>/dev/null | sort -n | tail -30

Find large files

find /opt -xdev -type f -size +100M \
-exec ls -lh {} \; 2>/dev/null | sort -k5

Don't use sort -h if the installed sort doesn't support it.

Find deleted files still consuming space

lsof +L1

Check Cell services

service celld status

Check MS process

ps -ef | grep '[m]s'

Check CELLSRV process

ps -ef | grep '[c]ellsrv'

Check RS process

ps -ef | grep '[r]s'

Check listening ports

ss -lntp

or:

netstat -lntp

Check MS error log

tail -100 \
/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log/ms.err

Check MS normal log

tail -100 \
/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log/ms.lst

22. Exadata DBA troubleshooting methodology

The biggest lesson from this incident isn't the individual command.

It's the order of investigation.

When CellCLI reports:

CELL-01519
Cannot talk to Management Server

don't immediately restart MS.

Use this sequence:

1. Check CellCLI error
       ↓
2. Check celld status
       ↓
3. Check MS process
       ↓
4. Check MS ports
       ↓
5. Check MS logs
       ↓
6. Check df -h
       ↓
7. Check df -i
       ↓
8. Find filesystem consumers
       ↓
9. Identify root cause
       ↓
10. Fix underlying problem
       ↓
11. Restart/recover service if necessary
       ↓
12. Validate CellCLI

This is far better than:

Error → Restart everything → Hope

23. Final takeaway

The most important distinction is:

CELL-01519
     ↓
NOT necessarily an MS-down problem

It can be:

MS process running
        +
MS unable to perform its work

In our case:

MS process              → RUNNING
CELLSRV                  → RUNNING
RS                       → RUNNING
HTTP listener            → RESPONDING
Filesystem `/opt`        → 100% FULL
MS write operation       → FAILED

Therefore:

A process being running does not prove that the service is healthy. Always check the service logs and the underlying OS resources.

Troubleshooting Exadata CellCLI Error: CELL-01519 / HTTP 404 Caused by /opt Filesystem Full

1. Environment

2. Initial Symptom

3. First assumption: Management Server is down

4. Check the MS process

5. Check listening ports

6. Test the HTTP endpoint

7. Check the Management Server logs

8. The actual error in the MS log

9. Check filesystem utilization

10. Check inode utilization too

General rule

11. Find what is consuming the filesystem

12. Inspect the raw storage files

13. Check for deleted files still consuming space

General troubleshooting principle

14. Check the partition layout

15. Check /etc/fstab

16. Check whether LVM is involved

17. Root cause

Primary root cause

Secondary symptom

18. Solution

19. What NOT to do

Don't delete raw disk files

Don't blindly resize the filesystem

Don't modify partitions blindly

Don't restart services blindly

20. After increasing /opt

Verify filesystem

Check MS logs again

Check Cell services

Test CellCLI

21. Useful troubleshooting command cheat sheet

Check filesystem capacity

Check inode usage

Check filesystem type

Check mount configuration

Check partition layout

Find large directories

Find large files

Find deleted files still consuming space

Check Cell services

Check MS process

Check CELLSRV process

Check RS process

Check listening ports

Check MS error log

Check MS normal log

22. Exadata DBA troubleshooting methodology

23. Final takeaway

Comments

Post a Comment

15. Check `/etc/fstab`

20. After increasing `/opt`